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ABSTRACT 


This  pape_*  presents  a  general  quantitative  system  for  association 
of  theory  and  observation.  The  system,  in  formulation  and  presentation, 
is  directed  toward  the  needs  of  the  user  investigator.  It  is  veil 
organized  for  use  with  automatic  machines  in  the  computing  and  informa¬ 
tion  processing. 

A  multivariable  distribution  approach  to  mod*il  construction  is 
used  in  accounting  for  errors  and  other  sources  of  variation.  As  usual, 
hypothesized  mathematical  descriptions  are  modified  in  accordance  with 
the  observational  data.  The  requirement  of  a  state  of  control  ia  re¬ 
garded  as  fundamental.  The  question  of  unidentif lability  is  given 
prominent  consideration;  and  in  this  connection,  the  general  necessity 
of  calibration  is  established  and  emphasised.  The  principle  of  maximum 
likelihood  is  suggested  as  the  most  acceptable  ranking  criterion  for 
the  system;  but  modifications  or  decision-theoretic  extensions  are  not 
precluded.  In  short,  the  ordinary  restrictive  conditions  are  not  imposed 
in  this  system. 

The  fundamental  ideas  are  discussed  within  the  contextual  framework 
’*  •  he  system.  The  general  principles  for  applying  the  system  are  pre¬ 
sented  and  discussed.  The  most  important  classes  of  models  are  deal's 
with  Mathematically  in  detail. 
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SIJUARY 


The  Problem 

We  are  concerned  here  with  the  problem  of  mathematical  model 
development  in  complex  observational  situations.  We  vis'n  to  adjust 
liypother.ized  mathematical  formulations  in  accordance  with  the  obnerva- 
tio.ial  data.  This  problem  is  of  course  basic  to  all  quantitative 
science.  Investigators,  whatever  their  specialized  disciplines,  are 
currently  confronted  with  a  bewildering  hodgepodge  of  scientific  metho¬ 
dologies.  To  the  great  majority,  preoccupied  with  their  individual 
specialties,  the  extensive  literature  of  potentially  applicable  mathe¬ 
matics,  statistics,  philosophy  of  science,  etc.,  will  always  bo  practi¬ 
cally  inaccessible.  Also,  those  techniques  which  have  ordinarily  been 
accepted  and  employed,  ir.  general  tend  to  embody  unrealistic  assumptions, 
inherent  inadequacies,  or  both. 

A  generally  valid  system  is  needed  which  is  reasonably  comprehen- 
siole,  at  least  in  application.  In  addition,  the  system  should  impose 
no  significant  inconvenience  on  the  user  investigator.  More  or  less 
generally  valid  systems  have  been  attempted  previously;  however,  re¬ 
strictions  on  their  use  or  problems  of  application  have  apparently  pre¬ 
cluded  their  acceptance. 

Results 

In  this  paper,  a  completely  general  system  for  mode]  development 
Is  presented.  Hie  entire  formulation  his  been  carried  out  as  an  attempt 
to  match  the  viewpoint  of  the  working  investigator  rather  than  ttet  of 
the  mathematical  statistician.  The  system,  in  all  of  its  component 
methodologies,  is  principally  directed  to  the  understanding  and  con¬ 
venience  of  the  investigator.  The  valid  association  of  thecry  and 
observation  is  accomplished  without  resort  to  restrictive  conditions. 
Except  for  the  most  general  and  fundamental  criteria,  such  as  the 
principal  of  maximum  likelihood  which  Is  herein  suggested  as  most 
appropriate,  any  needed  assumptions  are  regarded  as  solely  the  investi¬ 
gators  responsibility  and  as  necessary  parts  of  hie  initial  hypotheses. 
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•Rie  system  organisation  is  specifically  oriented  toward  use  in 
conjunction  with  large-scale  automatic  computing  systems.  It  is  pre¬ 
sumed  that  most  investigators  will  have  access  to  such  equipment  in  the 
near  future,  and  that  eventually  no  routine  mathematical  analysis  of 
significant  complexity  will  be  attempted  without  such  access.  Thus,  the 
matter  of  the  investigators  convenience  will  be  resolved. 


Fundamental  ideas  are  discussed  briefly,  within  the  contextual 
framework  of  the  system.  The  general  principles  for  applying  the  system 
are  presented  and  discussed.  The  most,  important  classes  of  models  are 
dealt  with  mathematically  in  detail.  It  is  hoped  that  this  paper  will 
also  serve  as  a  handbook  of  general  investigative  methodology. 
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INTRODUCTION 


This  paper  presents  a  general  quantitative  system  for  association 
of  theory  and  observation.  Most  phenomena  of  scientific  interest  are 
at  least  subject  to  the  possioility  of  quantitative  description.  Phe¬ 
nomena  subject  to  quantitative  description  are  also  suoject  to  compact 
mathematical  representation.  It  is  not  necessary  that  such  represen¬ 
tation  be  made;  a  complete  set  of  quantitative  observations  or  data 
points  contain  in  themselves  all  available  information,  by  definition. 
However,  it  is  our  faith  in  the  orderliness  of  creation,  together  with 
past  supporting  experience,  which  engenders  In  us  the  desire  to  fill 
in  the  space  between  the  data  points.  We  prefer  to  search  for  mathe¬ 
matical  descriptions  which  not  only  account  for  nil  past  observations, 
but  also  all  possible  observations,  in  order  that,  prediction  wi  11  be 
possible .  If  a  description  (or  model)  is  simple  and  elegant,  and  at 
the  same  time  serves  to  correctly  predict  a  reasonable  number  of  subse¬ 
quent  observations,  we  feel  that  understanding  of  the  subject  phenomenon 
has  been  achieved. 

'ftiua  our  stated  purpose  in  model  development  is  description  and 
n<  t.  necessarily  decisive  action.  It  is  of  course  Implied  that  d-.crip- 
ti^n  eventually  leads  to  decisions  and  actions,  but  actions  of  character 
an*,  purpoce  yet  to  be  established.  Specifically,  the  criteria  of  future 
decisions,  and  the  utility  values  of  the  possible  ultimate  consequences, 
are  at  present  unknown. 

The  decision -theoretic  approach  has  attained  greet  stature,  but 
not  every  Investigator  can  adopt  a  completely  behavioristic  philosophy. 
Probably  moat  will  remain  fundamentally  realists .  Nevertheless,  noth¬ 
ing  precludes  the  inclv3lon  of  decision  theoretic  considerations  into 
the  methods  of  this  pa?  >r. 

We  are  concerned  1  ire  with  the  problem  of  model  development  in 
complex  observational  situations.  In  such  cases,  simple  and  elegant 
descriptions  are  difficult  to  obtain.  Economic  pressure  or  urgent 
necessity  may  force  us  to  resort  to  oversimplifications  and  unjustified 
assumptions.  Unfortunately,  descriptions  obtained  under  such  conditions 
ere  not  likely  to  be  useful  in  prediction,  and  their  contribution  to 


understanding  Is  likely  to  be  small.  The  usual  strategy  In  dealing 
with  complex  problems  Is  to  attempt  a  division  Into  more  manageable 
subprobleme.  Hopefully,  the  pvhproblema  will  respond  to  the  available 
investigative  procedures.  liere  too,  the  results  Aie  not  always  success¬ 
ful. 

In  this  paper  an  opeiational  system  Is  proposed  for  use  in  develop¬ 
ment  of  complex  nflthematical  models.  To  this  end,  consider  tnat  com¬ 
plexity  in  observational  situations  can  be  conveniently  divided  into 
two  classes.  First,  the  complexity  mey  be  due  to  a  multiplicity  of  in¬ 
fluential  variables.  Second,  the  complexity  may  be  due  to  the  presence 
of  interacting  phenomena. 

In  the  first  instance,  if  the  number  of  influential  variables  is 
very  large,  it  may  be  best,  in  the  Judgment  of  the  investigator,  that 
the  variables  be  grouped  and  their  combined  group  influences  be  observed 
and  described.  For  example,  a  group  of  influences,  unobservable  indi¬ 
vidually, may  be  represented  in  combination  as  a  random  variable.  Or, 
if  the  number  of  variables  is  not  so  large  as  to  justify  grouping,  the 
investigator  might  not  wish  to  so  simplify  his  theory.  Instead,  in 
spite  of  the  complexity,  he  may  elect  to  describe  deterministically  the 
relationships  between  the  numerous  variables  taken  individually. 

In  the  second  instance,  it  may  be  the  case  that  two  or  more  dis¬ 
tinct  phenomena  are  so  interrelated  that  they  cannot  be  observed  in 
isolation.  For  example,  most  measurement  phenomena  are  subject  to 
interaction  with  those  phenomena  which  are  the  -rrors  of  observation. 
Also,  It  is  a  cannon  occurrence,  that  observations  can  only  be  made 
indirectly.  That  is  to  say,  the  unobservable  variables  of  primary 
Interest  can  only  be  determined  as  functions  of  variables  which  are 
directly  observable. 

Any  system  for  development  of  complex  mathematical  model  6  must  be 
sufficiently  general  to  encompass  these  situations.  In  the  first 
Instance,  means  must  be  provided  for  either  grouping,  or  individually 
relatii«,  the  influences  of  variables,  or  both  simultaneously.  In  the 
second  instance,  means  must  be  provided  for  separately  describing  the 
effects  of  Interrelated  phenomena.  The  system  herein  presented  is 
generally  applicable,  it  is  reasonably  simple  in  conception,  and  it 
lends  Itself  veil  to  use  of  automatic  machines  in  computation  and  in 
information  processing. 


The  need  for  an  organised  or  systematic  approach  to  model  develop¬ 
ment  has  become  '  easingly  evident  in  recent  years.  This  is  due  in 
part  to  the  acce.  .rating  trend  toward  greater  complexity  of  research 
problems  In  all  of  the  quantitative  sciences.  But  in  addition,  a  vide 


2 


/ 


intractable  vlth  currently  available  methods.  New  methods  are  intro¬ 
duced  here  which  have  resulted  in  the  solution  of  the  previously  intract¬ 
able  problems,  and  which  at  the  same  time,  have  provided  a  basis  for  the 
organisation  of  model  development  which  is  presented  in  this  paper. 

It  is  perhaps  helpful  to  mention  some  major  fields  of  research 
which  will  benefit  from  use  of  this  system.  ?or  example,  in  the  bio¬ 
logical  sciences,  reaction  rate  models  are  assuming  increasing  import¬ 
ance,  and  stochastic  models  are  more  widely  employed  than  ever.  The 
large  number  of  variables  and  interacting  phenomena  involved  in  bio¬ 
logical  systems  has  resulted  in  a  generally  unsatisfied  demand  for 
accurate  mathematical  models.  Operations  research  is  concerned  almost 
exclusively  with  mathematical  model  development.  With  reference  to 
these  models,  decisions  are  frequently  nade  which  are  of  great  and 
immediate  importance  to  society;  consequently,  the  risk  in  resorting 
to  inadequate  or  inaccurate  models  ie  large.  The  social  sciences 
generally  are  in  need  of  accurate  ana  useful  mathematical  models  for 
complex  phenomena.  For  example,  econometri  c  ane  lysis  has  been  hampered 
by  the  lack  of  adequate  theory  ir;  model  construction;  although  consider¬ 
able  progress  has  been  made  in  this  field  with  linear  and  polynomial 
models.  Also,  th-.-  physical  sciences  are  no  less  influenced  by  the 
trend  toward  research  interests  in  complex  phenomena.  The  relatively 
recent  introduction  of  multidimensional  pulse-height  analyzers  (high¬ 
speed,  digital,  multivariable  data  point  recorders)  ic  intensify* ng  the 
need  for  precise  methods  in  data  interpretation. 


At  this  time,  it  is  not  inappropriate  to  suggest  "at  a  reasonable 
degree  of  standardisation  be  introduced  in  scientific  method  and  report¬ 
ing.  Potentially,  this  can  occur  as  a  useful  byproduct  of  the  system 
application.  In  some  circumstances,  where  problems  in  research  manage¬ 
ment  exist,  the  basis  may  be  provided  thereby  fcr  control  of  research 
quality.  But  most  useful  perhaps,  is  the  improvement  In  communication 
which  may  result.  It  should  be  possible  to  readily  compare  or  casbine 
models  developed  under  widely  varying  research  environments .  Finally, 
use  of  generally  accepted  procedures  together  with  automatic  data  pro¬ 
cessing  can  ser/ :  to  free  soarce  professional  investigators  from  many 
routine  or  peripheral  aspects  of  research. 


tround  Note 


As  stated,  the  general  problem  with  which  we  are  concerned  Is  that 
of  the  meaningful  association  of  theory  and  observation.  Bare,  a  prac¬ 
tical  user-oriented  system  bar  been  presented  for  the  development  at 
mathematical  models.  The  calculus  of  probabilities  and  the  Principle 
of  Maximum  Likelihood  have  been  employed.  First  to  apply  the  calculus 
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of  probabilities  for  these  purposes  was  Thomas  Bayes.  Carl  Friedrich 
Gauss  was  first  to  formulate  and  apply  the  Principle  of  Maximum  Likeli¬ 
hood  in  1809  (see  P.  A.  Fiolier  (4),  pp.  20-21,  and  H.  F.  Trotter  (l3)» 
pp.  127-).  Gauss  approached  the  general  problem  from  the  point  of  view 
of  his  least  squares  theory. 

In  the  interim,  the  completely  general  problem  has  been  rarely  con¬ 
sidered,  at  least  from  the  practical  point  of  view.  On  the  other  hand, 
restricted  problems  have  been  intensively  investigated.  In  particular, 
methods  assuming  linearly  related  or  normally  distributed  variables 
dominate  the  literature.  These  of  course  include  such  approaches  as 
analysis  of  variance,  correlation  analysis,  and  regression  analysis. 

With  respect  to  model  development  under  restricted  conditions,  linear 
and  polynomial  regression  have  found  wide  acceptance.  However,  a  per¬ 
sistent  interest  has  been  maintained  by  a  relatively  small  number  of 
investigators  in  the  more  general  problem  of  more  than  one  variable 
subject  to  error.  Many  techniques,  valid  for  restricted  conditions, 
have  been  developed.  Excellent  reviews  of  this  work  are  offered  by 
M.  G.  Kendall  (6),  and  Albert  Madansky  (3). 

Several  attempts  to  provide  user-oriented  systems  stand  out. 

W.  E.  Deming  (3),  in  1943,  presented  a  general  system  based  on  the 
principle  of  least  squares.  Many  somevtot  less  general  expressions 
of  the  least  squares  theory  have  been  successfully  reduced  to  practice 
using  digital  computers.  For  example,  R.  H.  Moore  and  R.  K.  Zelgler 
(10),  i960  were  among  the  first  to  develop  a  useful  and  well-documented 
program  for  generalised  non-linear  regression. 

The  most  significant,  as  veil  as  the  most  general,  approach  pre¬ 
viously  offered  is  probably  thatof  Trygve  Haavelmo  (5),  in  1944.  The 
treatment  is  thorough,  it  is  a  system  in  the  sense  of  this  paper,  and 
it  is  oriented,  at  least  in  spirit,  to  the  needs  of  the  user  investigator 
(in  this  case  econometricians ) .  Also,  the  mathematical  formulation 
constitutes  at  least  one  or  two  additional  model  types  (which  are  not 
included  in  this  paper).  However,  the  formulation  is  quite  complex  and, 
as  acknowledged  by  the  author,  appears  not  to  be  readily  manageable  in 
practice,  (for  example,  tvo  nonlinear  transformations  and  tvo  integra¬ 
tions  are  generally  required  for  derivation  of  the  observation  model 
hypothesis.)  Also,  the  initial  hypotheses  required  nay  be  much  more 
appropriate  to  economic  theory  than  to  that  of  other  disciplines. 
Nevertheless,  it  is  unfortunate  that  this  work  faes  not  received  more 
widespread  attention. 

In  this  paper,  ve  have  particularly  aspired  to  simple,  easily 
acceptable,  and  manageable  formulations  for  all  model  types.  For 
example,  the  "Intersection-projection"  transformation  of  this  paper, 
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effected  by  substitution,  is  essential  to  simple  formulation  vhere 
models  involve  structural  relationships.  This  device  seems  obvious 
enoi^h,  as  will  become  apparent;  it  is  frequently  applied  automatically, 
without  specific  reference,  in  trivial  cases.  However,  it  has  apparently 
not  been  used  previously  in  general  formulations  ocf  the  type  here  under 
consideration. 

In  this  paper,  it  is  emphasized  that  distribut-*  ons  need  not  be 
random;  aind  here  also,  the  proper  use  of  systematic  distributions  is 
Indicated.  A  similar  view  on  the  concept  of  random  selection  is  expres¬ 
sed  by  R.  B.  Braithwaite  (l),  1953-  However,  there  seems  to  be  no  pre¬ 
vious  recognition  in  the  literature,  and  certainly  no  emphasis,  that 
existent  systematic  distributions,  and  in  particular  purposeful  syste¬ 
matic  distributions  (as  in  experimental  designs),  are  appropriate  and 
frequently  necessary  elements  in  formulation  of  concordant  sets  of 
hypotheses. 

The  definitive  consolidation  of  work  on  unidentif lability  was  given 
in  a  paper  by  T.  C.  Koopmans  and  0.  Reiers^l  (7),  in  1950.  They  begin 
with  recognition  of  the  fact  that  a  completely  general  formulation  of 
the  problem  of  statistical  inference  must  encompass  not  only  observable 
populations,  but  in  consideration  of  errors  of  observation  or  other  dis¬ 
turbances,  the  theory  or  structure  of  the  underlying  true  phenomena 
which  is  thought  of  as  generating  the  observed  distribution  must  also 
be  considered.  The  problem  of  identification  is  then  said  to  be  that 
of  drawing  inferences  from  the  probability  distribution  of  observed 
variables  to  the  underlying  structure.  This  formulation,  by  Koopmans 
and  Relers^l,  may  be  thought  of  as  constituting  a  non -geometric,  non - 
operational  statement  of  the  Basic  Principle  of  the  present  system. 

Both  expressions  are  closely  related  to  the  classic  view  on  the  influ¬ 
ence  of  error  as  expressed  by  Gauss  (13),  pp.  1,2:  that  errors  are 
either  constant  (systematic)  or  irregular  (random),  and  that  the  con¬ 
stant  error  cannot  be  estimated  from  the  observations. 

In  addition  to  the  above  introductory  contribution,  Koopmans  and 
Relersfil  go  on  to  express  in  greater  detail  the  general  concept  of 
identification  and  to  give  examples.  Their  paper  makes  two  other  points 
which  are  pertinent  here.  First,  we  are  warned  (pp.  169- 170)  against 
the  temptation  to  specify  models  in  such  a  way  as  to  force  identlf labil¬ 
ity,  since  scientific  honesty  deasnds  that  specification  of  a  model  be 
based  on  prior  knowledge  of  the  phenomenon  studied.  Second,  they  point 
out  (p.  179)  that  even  if  all  parameters  are  not  Identifiable,  it  re¬ 
mains  possible  to  construct  identifiable  functions  of  these  parameters 
which  constitute  useful  scientific  Information.  Thus,  the  appropriate 
direction  of  subsequent  work  would  seem  to  be  clearly  indicated. 
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In  view  of  the  above  background,  it  then  becomes  relevant  to  ask 
why  c\  workable  and  acceptable  system  of  adequate  generality  has  not 
been  previously  constructed.  Several  possible  explanations  come  to 
mind.  Principally,  the  general  necessity  of  resorting  to  specially 
constructed  calibration  models  and  ultimately  to  actual  calibration  in 
order  to  xeeolve  problems  of  unidentif lability  in  complex  observational 
situations  has  apparently  not  been  recognized.  The  emphasis  in  the 
literature  lias  been  on  determination  of  those  conditions  necessary  for 
complete  identification  in  restricted  situations.  In  this  connection, 
refer  again  to  Madansty  (8).  There  appears  to  be  no  recognition  that 
the  information  required  to  verify  such  conditions  must  be  obtained  in 
most  cases  by  actual  calibration.  Further,  any  other  means  of  obtain¬ 
ing  the  needed  information  constitutes  effective  calibration.  Thus, 
it  is  perhaps  a  principal  contribution  of  this  paper  that  calibration 
(or  effective  calibration),  long  regarded  in  classical  error  theory 
as  an  essential  feature  in  scientific  method,  as  applied  in  simple 
measurement  situations,  is  here  established  as  also  essential  in  com¬ 
plex  observational  situations.  Complex  phenomena  are  of  course  widely 
employed  as  "controls"  in  experimentation;  but  these  are  trivial 
instances  of  calibration. 

This  does  not  entirely  explain  the  apparent  premature  abandonment 
of  generality  in  most  of  the  literature.  It  is  possible  that  preoccupa¬ 
tion  with  pencil  and  paper  analytic  tractability  has  played  a  part. 

Thus,  it  is  appropriate  to  point  out  the  advantages  of  computer-oriented 
mathematics,  particularly  in  avoiding  problems  imposed  by  the  limitations 
of  the  notation. 

As  is  well  known,  the  concept  of  state  of  control  is  due  to  W.  4. 
Shew hart  (12)  1939*  Suitable  Introductory  reading  is  provided  by 
Munroe  (ll),  Mood  (9),  and  Cram6r  (2). 


THE  BASIC  PRUfCIPIZ  OF  THE  SYSTEM 


For  purposes  of  this  system,  the  phenomenon  under  observation  is 
theoretically  represented  by  a  distribution  of  points  or  events  over 
the  Cartesian  space  of  all  variables  (herein  coordinate  variables) 
which  are  considered  to  be  influential  or  otherwise  of  Interest  in  the 
investigation.  It  la  demonstrable  that  such  a  representation  may  be 
made.  The  distribution  la  generally  hypothesized  in  the  contact  fora 
of  a  Joint  density  function,  either  over  the  >  ntlre  space,  or  over 
appropriate  subspaces.  The  exact  manner  in  which  such  representation 
is  accomplished,  for  the  most  likely  special  situations,  is  resolved  in 
this  paper.  The  ge rural  method  is  also  discussed. 
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Since  the  model  1b  a  density  function,  it  serves  to  specify  the 
distribution  of  weight  or  of  potentially  existent  events  over  the 

model  space.  In  general,  not  all  arguments  of  the  d  ms ity  function 
are  mndcm  variables;  some  coordinate  values  may  be  selected  system* 
atically  in  accordance  with  a  prearranged  experimental  design.  In 
other  words,  generally  It  is  not  a  probability  distribution  which  is 
specif  lei,  if  by  the  term  "probability"  we  imply  randan  selection. 

Hot  all  variables  of  interest,  and  which  are  represented  in  the 
model,  are  also  observable.  Of  special  interest  are  the  errors  of 
observation  and  the  associated  hypothesized  true  but  unobservable  vari¬ 
ables  which  they  modify.  That  is  to  say,  we  distinguish  between  observ¬ 
able  variables  subject  to  the  error  of  observation,  and  the  associated 
unobservable  variables  in  fact,  which  would  be  free  uf  error  could  they 
bo  observed  directly.  Dencrtir^j  actually  observed  values  by  subscript  q» 
and  errors  introduced  in  the  process  of  observation  by  e,  the  relation¬ 
ship  is  as  follows: 

*0  =  X  +  ex 

In  this  situation,  all  three  variables  are  of  interest  and  all 
three  are  necessarily  included  among  the  coordinate  variables  of  the 
model  space  over  which  the  phenomenon  of  interest,  including  thoee 
aspects  of  the  phenomenon  which  are  exclusively  related  to  the  process 
of  observation,  is  hypothetically  represented  Thus,  there  exists  a 
subspace  of  those  variables  subject  to  observation.  To  these  we  now 
direct  our  attention. 

Hie  subspace  of  the  observable  variables  may  be  thought  of  as  the 
means  whereby  the  phenomenon  as  represented  by  the  model  is  revealed 
to  the  observer.  In  the  same  way  that  we  might  view  the  3-dimensional 
interior  of  a  house  through  a  window,  a  2 -dimensional  aperture,  the 
sub ipace  of  observable  variables  functions  as  a  window  through  which 
we  may  view  the  higher-dimensional  representation.  Indeed,  by  defini¬ 
tion,  no  other  avenue  is  available.  However,  it  should  be  noted  that 
what  is  observable  by  one  method  of  observation  (one  set  of  instruments, 
one  group  of  investigators,  etc.)  is  not  necessarily  observable  by 
another.  Thus,  the  sans  phenomenon  might  be  viewed  through  a  number 
of  different  windows. 

In  this  system,  the  method  of  viewing  consists  of  projecting  the 
event  msss  image  of  the  model  from  the  model  space  of  all  variables  of 
interest,  onto  the  space  of  observable  variables.  This  is  usually 
accomplished  by  integrating  out  the  unobservable  variables.  That  Is  to 

for  each  point  of  the  observable  space,  the  distributed  event  mass, 
as  given  by  the  joint  density  function  model,  is  summed  for  all  values 


7 


of  the  unobservable  variables.  The  resultant  Joint  marginal  density 
function  serves  to  specify  the  distribution  of  potentially  obsermble 
events . 

A  precise  analogy,  which  is  at  the  same  time  an  actual  exjun,  ’?, 
is  given  by  the  process  of  X-ray  photography.  Here  the  model  is  3- 
dipens ional  and  consists  of  the  object  being  photographed.  Hie  observ¬ 
able  sjece  is  a  plane,  the  essentially  2-dimensional  X-ray  film.  The 
density  function  over  the  observable  space  is  given  by  the  distribution 
of  precipitated  silver. 

The  basic  principle  of  the  system  should  be  emphasised. 

Principle:  The  distribution  of  apparent  events  over  the  space  of 
observable  variables  constitutes  cur  total  knowledge  of  the 
real  unobservable  universe.  The  apparent  distribution  is 
literally  a  projected  image  of  the  distribution  of  real 
events  over  higher  unobservable  spaces. 


ASSOCIATION  OF  THBOfflf  AND  OBSERVATION 


As  additional  foundation  for  construction  of  the  system,  it  is 
presumed  that  quantitative  scientific  knowledge  is  best  advanced  by 
the  following  sequence  of  operations: 

1.  Consider  past  theory  and  observation. 

2.  Formulate  new  and  generally  incomplete  hypotheses  or  models. 

3.  finploy  observational  data  in  completing  the  one  or  more  formu¬ 
lated  models. 

k.  Employ  additional  observational  data  in  evaluating  and  compar¬ 
ing  the  completed  Kdels  or  theories. 

In  the  event  that  complete  models  are  hypothesized,  the  third  operation 
may  be  dispensed  with.  However,  this  is  not  generally  advisable,  since 
in  many  if  not  most  cases,  such  a  procedure  would  oe  tantamount  to 
guesswork. 

It  is  presumed  that  the  Joint  density  function  model  will  not  be 
completely  specified  buz  that,  in  consideration  of  past  theory  and 
observation,  only  the  mathematical  form  of  the  model  will  be  hypothe¬ 
sised.  That  is  to  say,  the  hypothesized  density  function  involves 
unknown  constants,  parameters  to  be  evaluated  in  consideration  of 
observational  data  not  influential  in  formulation  of  the  incomplete 
model. 
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In  this  system,  the  principle  of  maxima  likelihood  is  suggested 
as  being  most  generally  acceptable  for  the  purpose  of  evaluating  oar 
estimating  the  uncnovn  constants.  Given  a  set  of  observed  points  or 
selections,  the  individual  events  are  presumed  to  be  independent  in 
the  stochastic  senne.  Whether  ox  not  observations  or  selections  are 
random,  it  will  subsequently  be  seen  that  the  presumption  is  Justifiable. 
For  each  point  observed,  the  observable -event  density  is  expressed  as 
a  function  of  the  unevaluated  parameters.  The  product  of  all  such 
functions  is  the  likelihood  function,  defined  over  the  parameter  space. 
The  product  of  independent  event  densities  ordinarily  yields  the 
density  of  the  model  space  joint  event.  However,  as  a  function  of  the 
parameters  to  be  estimated,  it  is  said  to  express  the  likelihood  of 
the  parameter  space  joint  event.  The  maximum  likelihood  parameter 
estimators  are  given  by  the  coordinates  of  that  point  In  the  parameter 
space  for  which  the  likelihood  function  is  maximized.  However,  as  will 
be  seen,  these  estimates  are  not  necessarily  unique.  Fell owing  evalua¬ 
tion  or  estimation  of  the  parameters,  the  complete*!  model  may  be  com¬ 
pared  to  alternate  theories,  using  likelihood  as  the  ranking  criterion, 
and  the  same  data  in  seen  case. 


DESIGNATION  OF  THE  IHHiCMENON  DESCRIBED  BY  TK  MODEL 


Given  a  particular  density  function  model,  there  is  a  question  as 
to  whether  or  not  the  representation  has  meaning.  It  has  meaning  only 
to  the  extent  that  the  phenomenon  represented  can  be  otherwise  uniquely 
designated. 

Ve  speak  of  the  phenomenon  which  is  the  object  of  our  investigation, 
the  object  phenomenon,  as  er.  entity;  thus  we  Imply  that  its  essence  is 
unchanging.  In  a  sense,  we  presume  that  the  variation  represented  by 
the  astheaatl  cal  model  is  superficial  variation,  having  no  influence 
on  the  essential  character  of  the  object  phenomenon.  It  la  this  essen¬ 
tial  unchanging  character  which  must  be  uniquely  designated  if  the 
quantitative  description  of  variation  is  to  be  unambiguously  interpre¬ 
table.  It  is  therefore  reasonable  to  require,  in  addition  to  the 
specifications  of  the  mathematical  model,  a  characterisation,  consisting 
of  a  set  of  designating  statements  or  classifications  which,  whenever 
or  wherever  applicable,  serve  to  effect  the  valid  association  of  the 
object  phenomenon  and  its  mathsast leal  modal.  Thus,  by  definition,  the 
object  phenomenon  occupies  the  class  intersection. 

In  a  given  instance,  the  association  is  Invalid  to  the  extent  that 
the  model  fails  to  descxlbe  the  object  phenomenon  as  designated. 
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Assuming  that  the  model  parameters  are  so  selected,  according  to  the 
initial  data,  that  a  good  fit  of  the  model  to  the  observations  is  ob¬ 
tained,  it  cannot  be  said  in  a  later  failure,  whether  it  is  the  quanti¬ 
tative  hypothesis,  or  the  designating  classifications,  which  are  in 
error.  Prom  one  point  of  view,  the  quantitative  model  is  inadequate 
in  that  a  source  of  excessive  variation  exists  which  is  not  represented. 
Or  alternatively,  the  original  investigator  was  careless  or  unlucky  in 
that  he  failed  to  completely  dee  ignite  the  conditions  under  which  he 
made  his  observations. 

Prom  another  point,  of  view,  a  phenomenon  which  is  adequately  desig¬ 
nated,  or  which  is  reproducible,  is  necessarily  in  a  state  of  control. 

A  phenomenon  inadequately  designated,  or  unreproducible,  is  out  of 
control  by  definition.  However,  in  tlie  same  sense  that  not  all  vari¬ 
ables  of  interest  are  observable,  not  ell  significant  classifications 
are  obvious.  Where  a  state  of  control  is  not  existent,  it  is  theore¬ 
tically  up  to  the  investigator  as  to  whether  or  not  he  will  attempt 
to  account  for  the  unrepresented  variation  by  altering  the  form  of  the 
mathematical  model,  or  by  placing  further  restrictions  in  the  form  of 
classifications  cn  the  phenomenon  to  which  the  model  applies,  and 
thereby  narrowin';  the  sc  >pe  of  the  investigation. 

It  is  generally  the  ronsatisfactory  solution  to  correct  a  poor  fit 
by  narrowing  the  scope  of  the  investigation.  It  is  better,  if  possible, 
to  interpret  a  poor  fit  as  specification  error  in  the  form  of  the  mathe¬ 
matical  model.  In  practice  however,  the  Investigator  may  be  deprived 
of  the  choice  which  is  rightfully  his  in  theory.  If  the  phenomenon  is 
not  in  a  state  of  control,  that  is  to  say  net  reproducible,  then  it  is 
possible  that,  a  new  mathematical  model,  so  specified  in  form  as  to  ade¬ 
quately  describe  the  observed  variation,  will  not  be  useful  for  the 
original  purposes  of  the  investigation. 

Prom  the  above  discussion,  two  things  are  apparent.  Pirst,  it  is 
essential  that  models  be  reexamined  under  the  widest  possible  range  of 
conditions  within  the  limits  imposed  by  the  classifications.  In  prac¬ 
tice,  experience  with  phenomena  similarly  classified  may  Justify  some 
relaxation  of  this  requirement.  Second,  the  cere  taken  in  classifying 
the  object  phenomenon  and  In  writing  the  designating  statements,  must 
correspond  to  the  effort  expended  in  developing  and  establishing  the 
quantitative  theory.  '3x»se  remarks  apply  with  particular  force  in  the 
application  of  this  system,  since  in  its  use  we  aspire  to  a  high  order 
of  discrimination. 

In  conclusion,  at  least  one  fundamental  condition  oust  apply  in 
use  of  the  system.  We  require  In  all  caaes  that  the  object  phenomenon 
aa  designated  be  reproducible  or  In  a  state  of  control.  More  precisely, 
we  require  that  the  finite  set  of  observed  events  which  are  actually 


obtained  constitute  a  representative  sample  of  a  potentially  infinite 
population  of  observations;  specifically  that  population  which  ia 
uniquely  associated  vlth  the  object  phenomenon  as  designated. 


DISTRIBUTIONS  MAY  RESULT  FROM  EITHER  RANDOM  CR 
SYSTEMATIC  SELECTION 


It  has  been  stated  that  in  this  system  variables  need  not  be  ran¬ 
domly  distributed  in  order  to  be  described  by  a  density  function. 

Before  this  idea  is  put  to  use,  it  should  be  discussed  in  some  detail. 

In  a  given  observational  situation,  the  values  of  a  given  variable  may 
be  either  purposefully  selected,  or  they  may  occur  as  a  natural  conse¬ 
quence  of  the  pneromonon  under  investigation;  however,  in  either  case 
they  may  be  either  .-andaaly  or  systematically  distributed. 

The  idea  of  randan  selection  is  historically  associated  vlth  the 
idea  of  a  lottery,  or  blind  selection  after  mixing  of  objects  distin¬ 
guishable  only  by  sight.  The  important  characteristics,  for  the  purpose 
of  this  discussion,  are  first  that  the  objects  are  contained  and  conse¬ 
quently  the  distribution  is  stable,  and  second  that  mixing  is  employed. 

To  say  t'nat  the  distribution  is  stable  is  to  say  that  a  state  of  control 
exists. 

The  mixing  idea  is  related  to  a  subjective  interpretation  of  pro¬ 
bability  as  degree  of  reasonable  belief.  Participants  in  the  lottery 
agree  that  it  will  be  fair  if  their  mutual  ignorance  of  future  selec¬ 
tions  is  assured;  hence  mixing  prior  to  selection,  or  "random"  selection, 
is  required. 

If  we  also  impose  the  requirement  of  replacement  following  each 
selection,  then  the  population  of  selections  becomes  potentially  infin¬ 
ite  and,  in  consideration  of  the  m'xlng,  the  individual  selections  are 
stochastically  Independent.  Thus,  at  least  In  this  case,  the  usual 
acceptable  conditions  are  established  for  construction  of  a  simple 
likelihood  function. 

However,  in  valid  application  of  the  system  we  require  that  the 
object  phenomenon  as  designated  exist  in  a  state  of  control.  This 
being  the  case,  the  ultimate  finite  set  of  selections  (or  sample)  ob¬ 
tained  is  representative  by  definition.  The  additional  requirements  of 
randan  or  stochastically  independent  selection  are  only  Incidental  meant* 
to  attainment  of  that  which  is  already  provided.  Once  we  are  in  pos¬ 
session  of  a  representative  sample,  the  means  of  its  attainment  are 


11 


immaterial.  We  can  arbitrarily  treat  the  individual  selections  as  hav¬ 
ing  been  obtained  under  conditions  of  stochastic  independence,  (and 
form  the  likelihood  function  accordingly)  whether  or  not  such  was  the 
case.  Specifically,  systematic  or  ordered  selection  is  not  invalidated, 
since  a  representative  sample  can  be  reordered  or  mixed  without  affect¬ 
ing  either  its  representative  character  or 'the' ultimate  maximum  likeli¬ 
hood  estimators.  However,  there  is  more  to  be  said. 

It  is  interesting  to  consider  some  advantages  and  disadvantages 
with  respect  to  random  versus  systematic  selection  in  those  situations 
wherein  the  investigator  has  the  choice,  that  is  where  selection  is 
purposeful.  A  principal  advantage  in  random  selection  lies  in  the 
simplification  of  the  model  that  is  its  frequent  accompaniment.  There 
may  be  variables  which  are  influential  in  the  object  phenomenon,  but 
which  are  not  of  interest  in  the  investigation.  The  effects  of  the  un¬ 
wanted  variation  may  be  discounted  by  uniform  randan  selection  of  ob¬ 
servations  over  the  space  of  the  unwanted  variables.  For  example, 
suppose  an  investigator  wishes  to  estimate  the  mean  moisture  content 
of  a  carload  of  wheat.  Suppose  that  the  car  is  known  to  have  been 
loaded  in  several  batches,  thereby  imposing  an  internal  stratification 
of  unknown  structure  on  the  carload.  If  the  structure  is  of  interest, 
then  the  spatial  coordirete  variables  should  be  included  in  the  model, 
along  with  hypotheses  concerning  the  form  of  the  interfaces.  Nothing 
is  wrong  with  this  approach,  except  that  the  model  is  unnecessarily 
complicated.  In  practice,  observations  would  probably  be  taken  at 
random  over  the  volume  of  the  car,  from  which  the  sample  mean  moisture 
content  would  be  computed. 

fhe  question  may  now  be  asked  as  to  whether  a  systematic  uniform 
selection  of  observations  over  the  car  would  not  be  better.  To  ade¬ 
quately  cover  the  volume  with  evenly  spaced  observations,  which  are 
sufficiently  close  together  so  as  not  to  miss  or  otherwise  give  improper 
weight  to  the  individual  strata,  would  probably  require  more  observations 
than  would  be  normally  required  for  a  random  sample.  In  another  situa¬ 
tion,  cyclic  effects  might  be  adversely  influential  if  the  systematic 
selection  is  in  phase.  However,  systematic  selection  can  also  be  de¬ 
signed  specifically  to  either  mask  or  detect  cyclic  or  other  systematic 
effects,  particularly  where  additional  information  is  available.  In 
fact,  where  economic  and  physical  factors  are  not  influential,  it  is 
conceivable  in  any  observational  situation  requiring  purposeful  selec¬ 
tion,  that  b  design  for  systematic  selection  of  observations  can  be 
found  which  is  superior  to  random  selection.  Of  course  economic  and 
physical  factors  do  intervene.  If  the  investigation  involved  selection 
of  representative  bowls  of  soup  from  a  large  container,  it  Is  certainly 
more  reasonable  to  actually  mix  the  soup  prior  to  ladling,  rather  then 
to  investigate  the  special  distribution  of  the  constituents  in  order  to 
achieve  the  desired  result  through  deduction. 
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There  is  surely  nc  virtue  in  ignorance  for  its  own  sake;  yet  ran¬ 
dom  selection  is  sometimes  used  when  there  is  no  compensating  advantage. 
Consider  a  distribution  of  purposefully  and  systematical ly  .^elected 
points  designed  to  closely  approximate  the  normal  or  Gauscian  distribu¬ 
tion.  If  selection  is  properly  done,  the  points  may  be  plotted  as  a 
smooth  histogram.  However,  the  distribution  of  the  same  number  of 
points,  selected  at  randan  according  to  the  normal  distribution,  and 
plotted  as  a  histogram,  generally  is  a  fairly  ragged  configuration.  In 
a  situation  where  the  selected  points  are  usea  in  an  estimation  process, 
and  wherein  confidence  in  the  estimate  is  increased  in  proportion  to  the 
extent  that  the  selected  point  set  is  actually  representative  of  a  nor¬ 
mal  distribution,  then  the  systematically  selected  point  set  is  to  be 
preferred.  The  additional  noise  introduced  in  random  sampling  can  only 
serve  to  increase  the  variance  of  the  estimate. 

In  any  case,  for  purposes  of  the  system,  r.b  distinction  is  made 
between  randomly  distributed  variables  and  syste:.  ..tically  distributed 
variables.  With  respect  to  the  concept  of  probability,  and  to  the 
extent  that  it  is  found  to  be  useful  in  applying  the  system,  probabili¬ 
ties  nay  be  simply  interpreted  as  objectively  determined  measures  of 
degree  of  reasonable  believabi.  ity  or  confirmation.  Tc  the  extent  that 
application  of  the  system  results  in  the  generation  of  such  measures, 
probability  can  be  said  to  oe  operationally  defined. 


MATHEMATICAL  THEORY  AND  DI3CUSSI0N 


In  the  remainder  of  this  paper,  it  is  the  purpose  to  first  discuss 
the  general  aspect  of  model  construction  in  the  context  of  the  system, 
then  to  investigate  seve  .1  useful  model  types  and  to  outline  the  speci¬ 
fic  otepe  in  their  construction,  and  finally  to  discuss  some  aspects  of 
implement*  t ion . 

In  the  usual  practical  application  of  the  system,  ar.  investigator 
is  expected  to  select  an  appropriate  standard  model  type.  For  example, 
if  his  ultimate  goal  is  the  ability  to  forecast  a  single  randan  variable 
e-rent,  he  will  probably  cnooae  a  regression  model.  The  number  of  stand¬ 
ard  model  types  may  be  increased  without  limit,  but  only  the  most  useful 
type 8  need  be  catalogued.  Eventually,  there  should  be  a  large  number 
from  which  the  investigator  may  chooee. 

For  each  standard,  model  type,  a  computer  program  should  be  prepared. 
Having  designated  the  object  phenomenon,  formulated  the  hyp  of  :ses, 
specified  the  associated  mathematical  forms,  and  collected  hit-  uata,  the 
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investigator  employs  the  appropriate  computer  program  in  evaluating 
the  parameters.  Resort  to  automatic  machine  computation  and  informa¬ 
tion  processing  not  only  eliminates  the  drudgery  but  enforces  the  de¬ 
sired  degree  of  standardisation  and  objectivity  in  the  analysis. 
Ideally,  the  computer  should  yield  a  report  of  the  investigation,  an 
integrated  presentation  of  the  phenomenon  as  designated,  the  data,  the 
completed  hypotheses,  and  other  useful  inf ornkition. 


Initial  Hypotheses 

In  most  model  types  to  be  discussed,  ve  distinguish  between  the 
phenomenon  of  primary  interest,  namely  the  object  phenomenon,  and  that 
phenomenon  which  is  the  act  of  observation.  The  hypothesized  true  but 
generally  unobservable  variables  of  primary  interest,  the  object  vari¬ 
ables,  are  generally  denoted 

Xl'  X2  "  "  "»  xn 

The  mathematical  representation  of  the  object  phenomenon  is  referred  to 
as  "the  object  model".  Included  in  the  object  model  hypotheses,  as 
initially  specified  are: 

1.  Joint,  marginal,  or  conditional  density  functions  which  des¬ 
cribe  the  distribution  of  various  subsets  of  the  object  vari¬ 
ables. 

2.  Equations  relating  various  subsets  of  the  object  variables. 

(If  the  variables,  as  specified  free  of  observational  error, 
are  naturally  so  related,  the  equations  are  said  to  describe 
structural  relationships.  In  other  cases,  they  may  describe 
prediction  or  regression  relationships.) 

3.  Inequalities  relating  various  subsets  of  the  object  variables, 
(ftwse  relationships  serve  as  constraints  on  the  distribution 
of  object  variables.  They  specify  the  region  boundaries,  when 
such  exist,  over  vhlch  the  Joint  density  functions  are  defined. 
Consequently  they  are  part  of  the  density  function  specification 

k,  In  each  of  these  fum., ions  or  relationships,  unevaluated  para¬ 
meters  6  will  probably  be  involved;  in  which  case,  known  or 
hypothesised  constraints  on  the  9  are  also  included  in  the 
initial  specifications. 

In  most  cases,  the  object  variables,  being  unobservable,  are  speci¬ 
fied  free  of  observational  error.  Corresponding  to  the  individual  object 
variables, are  hypothesised  but  unobservable  errors  of  observation.  The 
errors  are  denoted 


The  mathematical  representation,  which  by  hypothesis  describes  the  dis¬ 
tribution  of  errors  of  observation,  is  referred  to  as  "the  calibration 
model".  This  distribution  of  errors  is  generally  unknown;  but  the  func¬ 
tional  form  of  the  Joint  error  density  function,  which  my  or  may  not 
Involve  unevaluated  parameters,  is  hypothesized.  It  is  usually  prefer¬ 
able  to  describe  the  error  distribution  as  a  function  of  the  associated 
object  variables.  Consequently,  the  calibration  model  is  generally 
specified  as  a  conditional  Joint  error  density  function,  an  example  of 
which  is  here  denoted 

*  (*!’-— >'n/V~-'V  *> 

Here,  y  is  used  to  denote  density  functions  generically,  while  the  spe¬ 
cific  function  is  unambiguously  identified  by  the  arguments.  6  denotes 
a  vector  of  parameters  to  be  evaluated. 

For  any  fixed  point  (xq, — -,xn),  the  associated  Joint  error  density 
is  given  by  the  calibration  model.  TOiua,  we  say  that  the  object  vari¬ 
ables  enter  into  the  conditional  error  density  function  parametrically; 
that  is  to  say,  they  are  treated  as  constants. 

It  is  appropriate  to  mention  at  this  point  that,  although  the  den¬ 
sity  functions  are  not  necessarily  probability  density  functions,  they 
have  the  same  mathematical  properties.  Given  a  density  function  \|r: 

1.  *  is  defined  as  event  mass  per  unit  of  the  distributed  variable 
space,  for  every  point  in  the  space  of  all  arguments,  including 
those  which  enter  parametrically . 

2.  t  ie  everywhere  positive. 

3.  Event  mass  Is  additive. 

4.  Total  event  mass  over  the  distributed  variable  space  Is  unit 
mass. 

We  are  therefore  freo  to  make  use  of  the  calculus  of  probabilities. 
Specifically  in  the  situation  before  us,  the  conditional  Joint  error 
density  function  may  be  expressed  as  a  ratio  of  two  Joint  density 
functions. 

*  <'i- — 'Vv— ->v*>  • 
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Here,  6  denotes  a  vector  of  parameters. 

The  observable  variables,  true  object  variables  which  are  masked 
or  modified  by  the  errors  of  observation,  are  defined  and  denoted  as 
follows : 

v  \  *  v  k  ■ 1-— 

We  refer  to  these  as  "the  error  equations".  The  term  "error"  is  under¬ 
stood  to  signify  any  influence  or  set  of  influences  which  serves  to 
mask  or  displace  the  value  of  an  object  variable.  For  example,  an  un¬ 
known  lag  associated  with  a  time  variable  may  be  thought  of  as  error. 

Ultimately,  we  are  interested  in  an  observation  model,  a  joint 
density  function  model  over  the  space  of  observable  variables.  In 
application  of  the  system,  we  propose  to  obtain  the  distribution  by 
projecting  event  mass  from  some  hyperspace  which  includes  the  cv'~ervable 
space  as  a  subspace.  In  theory,  it  makes  no  difference  which  hyper- 
space  is  used,  or  how  the  hyperspace  model  is  initially  specified. 
However  in  practice,  certain  procedures  are  more  generally  useful.  For 
example,  it  *s  found  that  Initially  transforming  or  projecting  the 
calibration  model  onto  a  space  which  includes  the  observe1  le  space  is 
a  generally  useful  procedure. 

The  Error  Transformation 

Consiier  the  Joint  density  function  y(e1,r—,en,Xi, — Event 
mass  is  transformed  as  probability  is  transformed;  that  is  to  say,  in 
accordance  with  t.ie  calculus  of  probabilities,  the  absolute  value  of 
the  Jacobian  of  the  transformation  is  used.  Using  the  error  equations 
as  the  equations  of  transformation,  the  transformation  is  linear,  end 
/j/  Is  always  plus  one.  Thus,  the  translonaation  is  effected  by  simple 
substitution  from  the  error  equations  into  the  density  function. 

♦(®1~~ ,*n,' .  '**»)  *  '^(x0l”x1)' — '(xOn"xn)»xi>  — »xnJ 

“  ♦(x0p;  >xOn'xl»  ,xn) 

Dividing  both  sides  of  the  resulting  equation  by  the  marginal  density 
♦(x^,— -,xn),  we  obtain  the  desired  conditional  density  function. 

The  transformation  5 denoted  as  follows; 

♦(*!>-— ^nAi,—  ,Xh)  -  ♦(x01,---,x0n/xi--->xn)- 
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The  geometry  of  the  transformation,  for  one  object  variable,  is  shown 
in  Figure  1. 

Surface  Density  Functions 

Another  generally  useful  technique  in  derivation  of  the  observa¬ 
tion  model  hypothesi'  consists  of  eliminating  by  substitution  variables 
which  enter  parametrically  into  conditional  density  functions.  The 
substitution  is  valid  if  the  variable  to  be  eliminated  is  a  single- 
valued  function  of  other  variables  which  enter  into  the  function  para¬ 
metrically.  Geometrically,  the  effect  of  this  operation  is  in  two 
stages.  First,  the  joint  event  density  function  is  effectively  inter¬ 
sected  by  the  hypersurface  of  the  single  valued  function;  then,  the 
'■'ondlt icnal  event  mss,  which  is  thereby  distributed  over  the  hyper¬ 
surface,  is  effectively  projected  in  the  direction  of  the  variable  to 
be  eliminated,  onto  the  subspace  of  the  remaining  variables. 

For  example,  suppose  that  xp  =  F  (x2, — ,xn;  $).  This  is  a  struc¬ 
tural  relationship  between  object  variables.  We  also  suppose  that  F 
is  a  single-valued  function.  Here  again,  6  is  a  vector  of  parameters. 
Given  a  conditional  density  function  which  is  defined  for  all  fixed 

points  of  the  xp, - ,xn  space,  the  existence  of  the  structural  relation 

ship  exclusively  associates  the  density  function  with  points  on  the  F- 
surface,  which  is  imbedded  in  the  Xp, ,xn  space.  Since  F  is  single¬ 
valued  in  the  xp  direction,  the  density  function  can  be  mapped  one-to- 
one  (projected)  onto  the  X2,----,xn  subspace.  The  subspace  and  the  F- 
surface  are  of  the  same  dimensionality,  of  course.  It  is  not  recessary 
that  F  be  single -valued;  but  structural  relationships  can  usually  be  so 
specified,  and  a  mapping  which  is  not  one-to-one  can  result  in  a  signi¬ 
ficant  loss  of  information. 

If  the  conditional  density  function  is  the  previously  discussed 
calibration  model,  the  F-intersection  Is  executed  as  follows: 

,en/xl-  — ’V*>  ■  *(•!•—  VKv— ’V‘> 

- 

Functions  of  this  type  may  be  thought  of  as  generalized  or  "surface 
conditional  density  functions",  over  the  F-surface  imbedded  in  the 
original  ej,  — ,en,Xp,x2, space. 

tie  denote  this  transformation  in  either  of  two  ways:  either  to  the 
surface  conditional  ueusity  over  the  F-surface  imbedded  ?n  the  higher 
original  space, 


17 


*(ei'*"’en/xl’"'’xn;®)  -*■  ♦(«!>—  ,en/x2,— ,xn;e,*;F) 

or  if  no  confusion  results,  by  projection  to  the  ordinary  conditional 
density  over  the  subspace. 

»*„»•)  -  *(e1»— >en/xk,-— ,xn;0,4») 

The  geometry  is  shown  in  Figure  2. 

Concordant  Hypotheses 

We  now  consider  an  important  criterion  in  the  construction  of  any 
model.  It  is  necessary  that  the  model  components,  as  initially  specified, 
provide  sufficient  information  for  derivation  of  the  observation  model 
hypothesis.  In  general,  the  observation  model  is  a  joint  density  func¬ 
tion  over  the  space  of  observable  variables.  Consider  that  the  number 
of  ways  can  be  very  large  in  which  a  Joint  density  function  can  be  rep¬ 
resented  as  the  product  of  marginal  and  conditional  density  functions. 

For  only  three  variables,  there  are  thirty  ways.  Also  consider  that 
the  observation  model  joint  density  function  or  any  of  its  potentially 
numerous  factors  must  be  obtained  from  the  originally  specified  model 
components,  some  of  which  are  also  joint  density  functions,  by  an 
initially  unknown  series  of  transformations  or  projections  in  accordance 
with  the  system  concept.  Further,  the  conditions  of  the  investigation 
usually  predetermine  some  of  the  initial  model  specifications;  Imposed 
combinations  which  may  be  awkward  in  the  analysis.  Thus,  included  in 
the  general  problem  of  hypotheses  formation  is  the  not  generally  insig¬ 
nificant  problem  of  satisfying  the  sufficient  information  criterion. 

Overspecification,  or  too  much  information,  is  no  less  of  a  prob¬ 
lem,  since  it  can  result  in  an  inconsistent,  or  at  best,  an  inadequate 
model.  Thus,  a  set  of  initially  specified  model  components  is  required, 
vhicb  Is  sufficient  for  derivation  of  the  observation  model  hypothesis, 
and  which  is  also  necessary  to  the  goals  of  the  investigation.  Such  a 
set  will  be  called  a  "concordant  set  of  hypotheses".  A  principal  prob¬ 
lem  in  model  construction  is,  of  course,  selection  of  the  appropriate 
concordant  set. 

Available  Techniques 

Having  initially  specified  a  concordant  set  of  hypotheses,  the 
stated  immediate  goal  is  derivation  of  the  observation  model  hypothesis. 

In  general,  we  wish  to  combine  the  initially  specified  model  components 
in  such  a  way  as  to  eliminate  all  unobservable  variables  and  to  retain 
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Fig.  2  Illustrating  the  Intersection-Projection  Transformation 


♦(z/x,y;«)  StS»S1a.»(z/*;l»tF) 
*(z/x;e;F)  *£2l!££i22-,  *(z/xs9) 


all  observable  variables .  There  are  eight  manifest  techniques  which 
might  be  used  if  the  occasion  demands,  and  there  is  a  naturally  imposed 
order  of  precedence  in  their  use: 

1.  Exclude  the  unobservable  variables  from  consideration  in  the 
first  place.  This  presumes  that  their  influence  can  be  Justi¬ 
fiably  neglected;  otherwise  a  specification  error  is  introduced 
thereby. 

2.  Eliminate  unobservable  error  variables  by  the  one-to-one  error 
equation  substitution  transformation.  Of  the  eight  operations, 
this  induces  the  greatest  gain,  for  the  effort  expended. 

3.  Eliminate  fixed  or  "given"  variables  of  a  conditional  density 
function,  variables  which  enter  parametrically;  using  single¬ 
valued  functions'  of  other  given  variables  as  the  equations  in 
a  one-to-one  intersection-projection  transformation. 

k.  Eliminate  unwanted  variables  by  using  a  non-linear  one-to-one 
transformation;  mapping  into  a  space  of  variables  not  previ¬ 
ously  involved.  This  presupposes  the  prior  availability  or 
specification  of  the  appropriate  transformation  equations, 
equations  representing  monotonic  surfaces. 

5.  Use  a  more  complicated  variant  or  generalization  of  the  third 
technique,  whenever  the  available  relationships  are  not  single¬ 
valued.  It  is  necessary  to  partition  the  surfaces  into  regions 
of  single-valuedness,  and  to  sum  the  resultant  individual  pro¬ 
jected  event  densities.  This  substitution  transformation, 
being  many-to-one,  results  in  some  loss  of  information;  how¬ 
ever  in  proper  use,  this  would  be  the  necessary  loss  incurred 
in  viewing  the  hypotheses  from  the  limited  perspective  of  the 
observable  space. 

6.  Use  a  more  complicated  variant  or  generalization  of  the  fourth 
technique,  whenever  the  available  equations  represent  surfaces 
which  are  not  monotonic.  In  such  cases,  it  is  necessary  to 
partition  the  surfaces  Into  regions  of  monotonlclty,  such  that 
the  resultant  transformation  is  many-to-one,  specifically  not 
one-to-many.  The  resultant  individual  projected  event  densi¬ 
ties  are  summed  in  the  new  space.  This  is  ordinarily  the  most 
general  technique  applied  in  transforming  density  functions. 
However,  it  is  not  always  possible  to  avoid  the  one-to-many 
partitioning;  for  example,  the  transformation  surface  rjay  be 

a  hypsrsphere.  There  are  many  interesting  ways  of  dealing 
with  such  situations;  they  all  require  additional  hypotheses. 

In  the  case  of  the  hypersphere,  projected  event  mass  could  be 
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allocated  to  the  near  and  far  surfaces,  either  in  a  fixed  ratio 
or  in  a  ratio  which  is  introduced  as  an  additional  parameter 
to  be  evaluated.  Either  specification  constitutes  an  additional 
hypothesis. 

7.  Eliminate  variables  by  intersecting  a  joint  density  function, 
and  projecting  the  surface  distribution  of  Joint  event  mass 
onto  an  appropriate  subepace.  This  is  a  further  generalization 
of  the  third  and  fifth  techniques. 

8.  Eliminate  the  influence  of  unobservable  variables  from  consider¬ 
ation  by  integration.  Sum  the  event  mass  over  the  entire  space 
of  unwanted  variables,  in  order  to  consider  only  the  projected 
marginal  distribution  in  the  subspace  of  observable  variables. 

With  proper  application  of  these  techniques,  the  observation  model 
hypothesis  is  derived.  However,  the  observation  model  is  still  incom¬ 
plete,  as  are  the  object  model  and  the  calibration  model,  in  the  sense 
that  there  are  parameters  to  be  evaluated  or  estimated. 

The  parameters  are  the  6  and  4>  vectors,  of  the  object  model  and 
the  calibration  model  respectively.  There  are  of  course  other  ways  in 
which  parameters  may  be  introduced  in  the  hypotheses.  Some  will  be 
discussed;  they  are  here  denoted  |.  Now,  given  N  observations,  the 
data  are:  (xq^,  Xpgj, ----- jXQpj ) , U 1, -,N.  The  likelihood  function 
is  given  by 

N 

L  “  Y  ^Oll' 

The  likelihood  la  usually  naximlzed  by  maximizing  log  L.  That  is  to 
say,  a  point  Is  found  in  the  parameter  apace  of 0,i,  and  £,  subject  to 
possible  Initially  hypothesized  constraints,  for  which  log  L  attains 
Its  maximum  value.  Of  course,  N  must  at  least  equal  the  number  of  un¬ 
evaluated  parameters  In  order  that  they  be  determinate.  However,  we 
are  specifically  concerned  with  those  situations  wherein  observations 
are  not  In  short  supply  and  the  parameter  values  are  consequently  over- 
determined.  Thus,  In  the  presence  of  sampling  variation  and  specifica¬ 
tion  error,  the  parameter  values  must  be  estimated.  Actually,  many 
difficulties  can  occur  in  obtaining  the  maximum  likelihood  estimates, 
so  the  subject  Is  dealt  with  separately  later  on,  and  practical  methods 
are  Indicated. 

The  values  obtained  for  the  9,  ♦,  and  (  may  now  be  used  to  complete 
the  associated  models.  The  completed  object  model  may  now  be  se^  to 
provide  the  best  description  of  the  object  phenomenon  as  designated. 
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consistent  with  the  hypotheses  and  the  data,  according  to  the  principle 
of  maximum  likelihood. 

Also  consider  that  the  error  distribution  phencmenc-  can  be  the 
phenomenon  of  primary  interest.  The  maximum  likelihood  *  values  yield 
the  completed  calibration  model.  However,  in  cases  where  the  calibra¬ 
tion  model  is  constrained  by  the  object  model  hypotheses,  its  use  should 
be  suspect  for  points  in  the  space  of  object  variables  Which  are  not 
in  the  immediate  neighborhood  of  the  object  phenomenon  events. 

It  is  interesting  to  note,  in  application  of  the  system  generally, 
that  rounding  off  of  observed  values  to  the  number  of  digits  known  to 
be  or  suspected  of  being  significant  can  result  in  loss  of  information 
which  is  necessary  in  precise  evaluation  of  the  parameters;  in  particu¬ 
lar  where  errors  are  relatively  small  but  significant.  Observations 
should  probably  be  recorded  to  the  maximum  precision  obtainable  from 
the  observational  method  in  use.  This  is  clearly  contrary  to  existing 
practice  in  many  observational  situations. 

Purposeful  Distributions 

As  the  final  topic  in  this  general  discussion  of  model  construction, 
consider  that  it  is  frequently  necessary  to  purposely  impose  a  distri¬ 
bution  where  none  occurs  as  a  natural  consequence  of  the  phenomenon 
under  investigation.  The  distinction  between  purposefully  distributed 
variables  and  naturally  distributed  variables  is  useful  in  the  discus¬ 
sions  of  various  mor’el  types  to  follow.  This  distinction  is  solely  for 
the  convenience  of  the  investigator,  and  is  of  no  theoretical  or  philo¬ 
sophical  significance  not  previously  discussed.  ISie  act  of  purposefully 
distributing  or  weighting  a  subset  of  variables,  observable  or  otherwise, 
simply  imposes  on  the  investigator  the  requirement  that  the  distribution 
be  separately  specified  in  the  object  model.  An  alternative  is  to  con¬ 
sider  that  the  purposeful  distribution  is  a  stated  condition  which 
serves  to  designate  the  object  phenomenon.  This  last  procedure  neces¬ 
sarily  restricts  the  scope  of  the  investigation;  nevertheless,  it  is 
sometimes  appropriate.  Tor  example,  consider  the  situation  wherein 
there  are  variables  In  which  we  have  no  intexvst.  Then  the  object 
phenomenon,  as  designated,  may  reasonably  Involve  purposeful  uniform 
point  selection,  either  random  or  systematic,  over  the  space  of  the  un¬ 
wanted  variables,  In  order  to  simplify  the  model  specification. 

However,  in  general,  the  most  useful  procedure  Is  to  treat  the 
purposeful  distribution  as  a  separate  but  jonstltuent  phenomenon,  sepa¬ 
rately  specifying  the  fora  of  the  density  function  and,  If  necessaxy, 
evaluating  Its  parameters  either  separately  or  concurrently  with  the 
object  model  parameters.  Since,  by  definition,  the  functional  form  Is 
under  the  direct  or  indirect  control  of  the  investigator,  specification 
error  should  b  minimal  if  not  non-existent. 
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Purposeful  distributions  are  most  likely  to  be  directly  observable, 
although  this  is  not  necessarily  the  case;  therefore,  it  might  seem  most 
reasonable  to  specify  purposeful  distributions  over  the  space  of  observ¬ 
able  variables.  Unfortunately,  many  useful  object  model  components, 
taken  in  conjunction  with  a  directly  specified  observable  distribution, 
are  not  likely  to  form  a  concordant  set  of  hypotheses.  Consequently,  it 
is  frequently  preferable  to  specify  the  purposeful  distribution  over  the 
space  of  the  associated  object  variables.  An  important  exception  occurs 
in  regression  and  prediction  models,  where  che  specified  object  vari¬ 
ables  can  also  be  observable  variables . 


MODEL  TYPES 


Type  I  Model:  Simple  Measurement  with  Enrr 

In  this  first  and  simplest  of  all  observational  situations,  the 
phenomenon  under  investigation,  by  hypothesis,  is  represented  by  a 
single  point  - 0n)  in  space  of  object  variables,  the  coor¬ 

dinates  of  which,  tl»e  0j_,  are  the  unknown  parameters  to  be  evaluated. 
These  hypothesized  true  but  unobservable  values  are  masked  jy  the  errors 
of  observation.  In  other  words,  we  wish  to  perform  a  simple  act  of 
measurement  involving  error. 

For  clarity,  but  without  loss  of  generality,  the  system  is  illus¬ 
trated  first  for  a  Type  I  model  of  two  object  variables.  Thus,  the  six 
variables  of  interest  which  define  the  model  space  are:  x^,  Xg,  „he 
object  variables;  ej ,  eg,  the  corresponding  errors  of  observation;  and 
xoi,  x<)2>  the  observable  variables. 

The  object  model  is  given  by  the  equations: 

xx  ■  0^;  m  ®2 

These  equations  are  structural  relationships.  0j  and  0g,  are  the  object 
model  parameters,  constants  to  be  evaluated. 

The  calibration  hypothesis  is  appropriately  specified  as  a  condi¬ 
tional  Joint  error  density  function  as  follows: 


Again  ft  is  a  vector  of  unevaluated  parameters. 


Having  formulated  the  initial  hypotheses,  the  immediate  concern  is 
with  derivation  of  the  observation  model  hypothesis.  Tc  this  end,  the 
initial  hypotheses  must  be  combined  in  such  a  way  as  to  effect  elimi¬ 
nation  of  the  unobserwble  variables,  and  introduction  of  the  observ¬ 
able  variables.  Geometrically,  we  wish- to  project  the  combined -model 
event  mass  distribution  from  the  model  space,  or  its  subspaces,  mt'* 
the  subspace  of  observable  variab2.es.  First  the  error  equation  trans¬ 
formation  Is  applied. 

'Ke*1>e2/xi>x2:<1')  -  *(xoi’xo2/xi»x2;‘1'^ 

Thus,  the  joint  distribution  of  observable  values  is  given  for  any 
fixed  point  (xj,x2)  in  the  object  variable  space.  However,  by  our 
hypothesis,  the  object  model," -the  phenomenon  of  primary  interest.  1 c 
restricted  tc  a  single  point  in  the  x^,X2  space,  namely  (9]>62). 

Geometrically,  at  this  stage  of  the  derivation,  the  model  is  a 
joint  density  function,  defined  over  a  plane  which  is  imbedded  in  the 
b -dimensional  x01,x02>x: >*2  sPace’  The  plane  is  the  2-dimensional 
intersection  of  two  3 -dimensional  hyperplanes,  which  are  defined  for 
x1=61  and  x^62  respectively.  The  joint  density  function  is  given  by 

t(x01»x02* 

The  expression  does  net  involve  and  X2*  Consequently,  we  choose  to 
ignore  th®  ^-dimensional  environment  of  the  distribution;  but,  in  doing 
so  we  effect  the  desired  one-to-one  transformation,  or  projection  into 
the  2-dimensional  subspace  of  observable  variables.  Thus,  the  deriva¬ 
tion  of  the  lype  I  observation  model  hypothesis  is  accomplished. 

The  likelihood  function  can  row  be  formed,  and  the  parameter  esti¬ 
mates  obtained.  The  values  obtained  are  denoted  £2,  and  $.  01  and 

$2  complete  the  object  model.  The  $  values  obtained,  complete  the 
calibration  model.  However,  the  calibration  me*  cl  is  suspect  for 
points  of  the  object  variable  space  which  are  not  in  the  immediate 
neighborhood  of  $2) • 

In  conclusion  of  the  Type  I  model  discussion,  note  that  this  2- 
dimenwional  derivation  is  symmetrical  with  respect  to  the  object  vari¬ 
ables;  consequently,  it  extends  readily  to  any  number  of  object  vari¬ 
ables.  Per  simplicity  in  the  discussions  to  follow,  variables  which 
are  symmetrically  treated  in  the  derivation  ure  represented  as  vectors. 
In  fact,  all  model  types  ae  represented  are  expended  to  any  number  of 


25 


dimensions  in  consideration  of  the  indicated  variables  as  vectors. 

Thus,  the  Type  I  situation  and  that  of  other  model  types  are  also  pre¬ 
sented  without  explanation  in  a  more  compact  schematic  suitably  illus¬ 
trated  for  convenient  reference. 

In  the  Type  I  illustration,  the  conditional  error  distribution  is 
indicated  by  contours  of  equal  event  density  superimposed  on  the  space 
of  the  object  variables.  This  superimpos inion  so  serves  to  indicate 
the  appearance  of  the  distribution  of  observed  values  over  the  observ¬ 
able  space,  which  may  also  be  thought  of  as  superimposed;  and  appro¬ 
priately  in  this  case,  the  cooi*dinate  axes  coincide.  This  superimpo¬ 
sition  device  will  also  be  used  in  depicting  the  situation  for  other 
model  types. 

Type  II  Model:  All  Object  Variables  Distributed 

The  Type  II  model  is  characterized  by  the  fact  that  all  object 
variables  are  distributed  by  hypothesis.  That  is  to  say,  the  object 
model  involves  ape '.if  ication  only  of  a  Joint  density  function,  or  its 
marginal  and  conditional  density  function  factors.  The  Type  I  model 
is  actually  a  trivial  special  case,  wherein  the  specified  distribution 
assigns  all  of  the  event  mass  to  a  single  point. 

The  Type  II  illustration  shows  the  case  of  two  object  variables. 

The  initial  hypotheses  and  the  derivation  of  the  observation  model 
hypothesis  are  both  presented.  The  generality  of  the  vector  notation 
should  be  considered.  Any  number  of  Type  II  variants  are  admissible 
thereby. 

1.  x  (Xp-,^) 

2.  1r(x)  ♦(x1,—  ,*n) 

3.  ♦(xi,— ,xn)  need  no-c  be  specified  directly,  cs  a  single  Joint 

density  fui  ction;  but  nay  be  specified  in  factored  form.  For 
example;  #xk) .*(xk+1,—  ,xa/x1,  — -,xk) . 

k.  Bach  of  the  factors  nay  be  either  a  purposeful  distribution  or 
a  natural  distribution,  (in  an  investigation  devoted  exclusively 
to  calibration,  all  factors  might  be  purposefully  distributed.) 

Of  particular  interest  is  that  special  case  wherein  a  subset  x,  of 
object  variables,  is  observable  directly  without  error.  That  is  to  say, 
ex =0.  Let  y  denote  those  other  object  variables  which  are  subject  to 
errors  of  observation.  Then  the  initial  hypotheses  are  f(x,y ;6)  and 
*(ey/x,y;$)  ♦  (ex>ey/x>yJ4’)-  The  derivation  is  ns  follows; 
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TYPE  I  MODEL:  SIMPLE  MEASUREMENT  WITH  ERROR 


Object  Model  \  0 

Calibration  Model  v(e,l  X;  0) 

Derivation  of  the  Observation  Model  Hypothesis 

1.  -(ejx.0,1  —  i* (Xd I X;  <P) 

;J.  Mx0lx;  —  4(x0;  0,<t> ) 

Observations  x0l  (x0l)  ,xWl, - ,  i  1 

Likelihood 

-  N  '  ' 

L  H  4(Xo, ;  0,<P> 

t 

N 

log  L  =  2  log4(Xo,  ;  6, <t) 

I 

N  A  A 

(log  L) 

nu  2  log  4(Xq,  ;  6  ,<** 
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♦  (ey/x,y;<J>)  -  *(y0/x,y;$) 

>(x,y;6)  '^(y^x^jC)  -  ♦(yo,x,y;0,$) 

However,  x  x^,  so  that 

*(y0,x,y;e,4>)  t(yo,xo,y;0,4>) 

and  finally 

♦(y0>vy;  0'®)dy  =  ♦(xo'yo;e'®) 


Type  III  Mouel:  Structural  Relationships 

The  Type  III  model  object  variables  are  related  to  each  other  in 
the  initial  hypotheses  according  to  one  or  more  specified  equations. 

It  is  also  stipulated  here  that  the  variables,  as  specified,  do  not 
include  or  involve  the  errors  of  observation.  As  previously  stated, 
such  relationships  are  known  a3  "structural  relationships". 

The  relationships  are  presumed  to  be  single-valued  in  their  respec 
tlve  y  directions,  as  indicated  in  the  '.type  III  schematic.  It  <s  usu¬ 
ally  the  case  that  such  single -valued  representation  can  be  specified. 

In  the  intersection  operation  (Derivation  2),  whenever  more  than 
one  structural  relationship  is  involved,  it  is  of  course  most  expedi¬ 
tious  to  so  order  the  substitutions  that  no  variable  is  reintroduced, 
after  being  previously  eliminated. 

In  addition  to  the  specified  structural  relationships  and  the 
calibration  model,  a  Joint  marginal  distribution  ^r(x),  or  its  consti¬ 
tuent  factors  must  be  hypothesised,  as  shown  in  the  type  III  illustra¬ 
tion.  This  marginal  distribution  specification  is  necessary  in  order 
to  obtain  the  concordant  set  of  hypotheses.  In  the  usual  situation, 
we  would  expect  tliit  all  of  the  x  object  variables  would  be  purpose¬ 
fully  distributed;  but  this  need  not  be  the  case.  Components  of  this 
marginal  distribution  may  occur  naturally  and  be  of  interest  in  the 
investigation. 
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TYPE  II  MODEL:  ALL  OBJECT  VARIABLES  DISTRIBUTED 


Object  Model  v(x,  6) 

Calibrator  Model  v(exlx;V>) 

Derivation  of  the  Observation  Model  Hypothesis 

1.  </<e,|y;0)  —  <t (x<)  I x ;  <P) 

2.  <1  (x;  0)  ■  </(x$|  x;  4)  -  (x0,x; 

♦  «o 

3.  /_  ^  <|(x<,,x;  0,<tydx  -  i|(x0;  0,0) 
Likelihood 

N  A  A 

(lo-?  L)mai  -  i;  !<*  »(x0, ,  0,0) 

i 
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TYPE  III  MODEL:  STRUCTURAL  RELATIONSHIPS 


Object  Model 

1.  Joint  Marginal  Density  Function  i*(x,  £) 

2.  Structural  Relationships  y  -  F(x,0) 

Calibration  Model  ^(e,  ,ey|  x,y;  <£) 

Derivation  of  the  Ob»ervation  Model  Hypothesis 

1.  <'<ell.,eyl  x,y;  4)  —  ^(xo,y0!x,y;  <P> 

2.  4(Xfl,y0lx^,<^  —  ^(X0,y0lx;  6,0,  F) 

3.  <HxA)  ^(Xo,y0lx;  -  ^(x0,y0,x;  9A0 

4.  J  4'(x«.y0lx;  0,<f>,4)d*  ^  WxQ.yo'  O,0,i) 


Prediction 

Frequently,  there  is  less  Interest  in  the  theoretical  structural 
relationship  which  underlies  the  errors  odf  observation,  than  there  is 
interest  in  predicting  observed  values  yg  for  given  observed  values  Xq. 
In  such  cases,  the  usual  practice  Is  to  hypothesise  no  error  in  x.  Thus 

t(ex,ejr/x,y;®)  *(e^/x,y;*) 

1/Uy/x,  y;®)  -  +(y0/x,y;*). 

*(y0,x,y;4>)  -  +(y0/x;e,*;F) 

*(x;6)-*(y  Jx‘,0,*)  =  +(x,yo;0,$,O 


but  x  x^;  so 

The  prediction  function  is  then  given  by 

y0(xo^,$)  y0  ♦(yo/x0;M)iy0 

This  is  ordinarily  considered  to  be  a  kind  of  regression;  but  here 
we  call  it  "prediction".  Thp  name  "regression"  is  reserved  here  for 
cases  involving  no  underlying  structural  relationship. 

Type  IV  Model:  Regression  on  Observable  Variables 

In  those  analyses  wherein  all  variables  axe  distributed  under  the 
initial  hypotheses,  and  wherein  it  is  desired  to  describe  the  most  rep¬ 
resentative  or  mean  values  of  each  of  a  subset  of  observable  variables 
yQ  as  single-valued  functions  of  the  remaining  observable  variables 
x  Xq  (i.e.,  the  x  are  free  of  observational  error),  we  will  say  that 
the  development  is  a  case  of  "regress ion".  We  say  that  the  single- 
valued  functions  are  "regression  relationships",  niey  are  defined  as 
follows : 
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y0  =  F(xo,0)  =  y0  e)dy0 

The  necessary  constituents  of  the  object  model  are  a  Joint  condi¬ 
tional  density  function,  which  in  general  is  given  by  ^(y/x;0),  and  the 
structural  relationships  x  Xq  which  are  also  included  as  a  consequence 
of  the  model  type  definition.  These  necessary  constituents  together 
with  the  calibration  model  completely  determine  the  regression  rela¬ 
tionship  set  yo  =  F(xq,6).  Consequently,  the  problem  of  selecting  a 
concordant  set  of  initial  hypotheses  is  not  trivial.  The  derivation 
of  the  observation  model  hypothesis  is  shown  in  the  schematic. 

The  Joint  marginal  density  function  t(xQ;|)  of  the  directly  observ¬ 
able  subset  Xq  or  any  of  its  component  factors  may  or  may  not  be  of 
theoretical  interest.  Since  this  distribution  enters  into  the  observa¬ 
tion  model  simply  as  a  factor,  and  since  it  is  directly  observable  and 
therefore  may  be  purposefully  and  completely  specified  initially,  it 
(or  any  of  its  component  factors)  can  be  entered  into  log  L  as  an  addi¬ 
tive  constant.  As  such,  it  has  no  influence  in  the  maximization,  or 
ultimately  on  the  values  %  and  $. 

In  cannon  application  of  regression  analysis,  no  calibration  model 
is  specified;  there  is  no  error  in  the  y  variables  and  therefore 

♦(y/xQje)  =  t(y0/xQ,e) 

Also  commonly,  this  conditional  density  function  is  initially  hypothe¬ 
sized  for  a  single  yQ  variable  and  is  usually  specified  as  normal: 

*<V  %>  -  n  [p(ve),  VX(j) 

Further,  in  ordinary  application  of  regression  analysis,  the  observable 
Joint  aarginal  distribution  is  not  of  theoretical  Interest.  Also  °y0 
nay  be  thought  of  as  constant  and  Ignored.  Thus  any  specified  func¬ 
tional  form  for  the  regression  relationship  is  concordant,  and  the 
process  reduces  to  curve  fitting  according  to  the  principle  of  least 
squares: 
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TYPE  IV  MODEL:  REGRESSION  ON  OBSERVABLE  VARIABLES 


-*»Dl  *5 


Object  Model 


1 

Structural  Relationships  X  —  Xq 

2. 

Joint  Conditional  Density  Function  v(y!  X 

6) 

:y. 

Observed  Joint  Marginal  Density  Function 

v(Xq-  s) 

4. 

Regression  Relationships  yg  -  F(Xq,©)  - 

♦  • 

/  yo^Oo  |xo;0)dy 
—  *> 

Calibration  Model  4(e,  ,ey  1  x,y ;  4*  ~  if(ey|x,y; 

4* 

Derivation 

of  the  Observation  Model  Hypothesis 

1. 

4<ey|x,y;$)  —  4(y0|x,y;4*  -  </(y0l  *0,y ; 

2. 

4(y  1  x;  0)  —  f(y.lxo;  #) 

3. 

4<Xo;  4)  •  4(y|xe;0)  =  f(Xo,y;  0,4) 

4. 

^(xc,y;  0.0  •  ^(yol*c.y;  4*  =  H^,yo>y< 

0>.  0 

5. 

/_*%<x0,y0,y;  0,4>.Ody  =  4(xo,y0;  ®A0 

6. 

4(x0, yo;0A4)  =  4(x0;4  >  </Cvo|xo;0,^ 

Likelihood 

log  L 

=  ^  i  log  Ox0l ;  4)J  +  log  i/ <y o j  1  *ot 

.  0,4*] 
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(log  L) 


max 


or 


Q(log  L) 


min 


N 


.  V 

L> 

i 


[y01  -  F(x0,,  $)] 


0iJ 


for  constant  o 


yr 


Type  V  Model:  Discrete  Distributions 

In  general,  for  the  various  model  types  discussed,  the  indicated 
distributions  need  not  be  continuous.  They  may  be  specified  only  for 
a  finite  number  of  values  of  certain  of  the  variables.  Suppose  that 

certain  variables  x  =  (x,,- - ,xlt,---,xm)  by  hypothesis  assume  a  number 

q(k)  of  values  That  is  to  say,  for  each  x^, 

*k  =  ;  J  “  1»~~ 

In  the  manner  of  the  Type  I  model,  these  are  structural  relationships. 
However,  since  each  variable  assumes  a  multiplicity  of  values,  the 
individual  surface  distributions  of  event  mass  must  be  sunned  in  pro¬ 
jection,  according  to  the  fifth  of  the  previously  listed  available 
techniques. 

For  example,  consider  a  Type  n  model.  Suppose  that 
t(x,y;6,t)  -  t(x;t)*t(y/x;e) 

where 

♦(x;0  =  ♦(xjJlj)  ■  J  -  l, . ,q 

lere,  the  { j  are  the  discrete  set  of  x  values,  which  may  or  may  not  be 
known  initially;  and  the  tj  .  are  the  corresponding  set  of  marginal  den¬ 
sities,  which  eay  or  aay  not  be  known  initially.  Thus, 

t  indicating  implicit  parameters.  Referring  to  the  Type  n  schematic, 

we  also  have 


3^ 


*(x0,y0/x,y;*)  ♦(x0>y0/xJ,y;«) 

Consequently, 

♦(xo»yo>xA>y5e»<*,*Tij»i)  =  T»j,i'(y/xjie),t(x0*y0/xj>yj®) 

However,  ve  do  not  proceed  exactly  according  to  the  Type  II  derivation; 
given  the  q  structural  relationships,  which  geometrically  are  repre¬ 
sented  by  constant  hyperplanes,  the  situation  calls  for  a  series  of  q 
intersections  of  the  joint  event  mass  distribution  defined  by  the  above 
joint  density  function.  This  is  in  application  of  the  seventh  of  the 
previously  listed  available  techniques.  The  resulting  set  of  q  surface 
joint  density  functions  is  given  by 


,  J  -  l,— -,q. 

As  stated,  the  associated  projection  transformation  is  many-to-one  and 
the  densities  must  be  summed  in  the  subspace.  Hius  ve  proceed. 

*(xo»yo,xjiy;S,*,V*)  ■*  ♦(Vyo'y;0'*' 


where 


The  last  step  la  simply  to  project  into  the  observable  space: 


In  sane  situations,  it  is  expected  that  other  nonconstant  struc¬ 
tural  relationships  will  also  be  specified  in  the  manner  of  the  Type  II 
model.  The  effect,  in  such  cases,  is  simply  to  require  the  additional 
associated  intersection-projection  substitution  transformations  in  the 
derivation  process,  provision  for  which  is  indicated  in  the  type  V 
schematic.  Die  additional  structured  variables  are  denoted  Z.  Two 
specialised  Type  V  situations  are  shown  in  the  illustrations . 
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TYPE  V  MODEL:  DISCRETE  DISTRIBUTIONS 


«»nt,  4/*  fct 


Object  Model 

1.  Joint  Marginal  Discrete  Density  Functions 

<i(x:  t)  -  f(xr  i3)  =  t jj  ;  j 

2.  Associated  X  Value  Discrete  Set,  Constant  Structural  Relationships 

xk  =  \i  ;  j  =  i, — ,q(k) 

3.  Additional  Nonconstant  Structural  Relationships  Z  =  F(Xf  ,y  .tj) 

4.  Joint  Conditional  Continuous  Density  Function  *l(y  I  XJ(0) 

Calibration  Model  <p(e„e,,eglxJ,y,z; 

Derivation  of  the  Observation  Model  Hypothesis 

1.  V,  •  <P(y lx, ;  0)  =  <><x,,y;  M-j) 

2.  ^(e„ey,etlx,,y,z;  -  </(x<>,yo.zelxj.y.z;  ^ 

3.  <'(xo*)ro»*«ixi,y»z;  ^  "*  ^x«*yo.zoixj-y;  ^ 

4.  ^(x,,y;  #,»?,)  •  ^(xo^Ok^ol xi»y •  "  WWo.zo-xi-y!  6*<!>-«»Ti)) 

5.  ^(xo,y0.*o^i^:  -  l^(xo.yo.zo.y;  0.*.“V)j.ty 

1 

6-  'S  V(xo,y9,z0.y;  oAvrfi.t))  =  <Hxo.y<>.zo-y;  ®AW»T?.0. 

J=l 

♦  «o 

7.  /  ^(Xflty0lz0,y;  0,0,w,T?,4)dy  =  </'(x0,y0,z0;  OAw.l.' 

—  « 

q 

S  or  ^(Xfl.yo.Zo;  6,<P,u,v,0  =  2  *?,  ^(Xo.yo^o^j  =  4,  ;  *.*.«•>» 

)=> 


Attribute  Distributions 


The  generality  of  the  Type  V  model  is  considerably  extended  if  we 
consider  that  the  x  need  not  be  quantitative  variables  but  that  the 
discrete  ''values"  they  assume  may  be  qualitative.  That  is  to  say,  the 
xj  may  represent  attributes  or  classifications,  of  which  the  relative 
likelihoods  or  weights, the  r^are  under  investigation. In  such  a  case, 
the  are  merely  implicit  qualitative  parameters.  Thus,  the  observation 
model  iypothesis  is  given  by 


+v*,yr 


,z0;6  ,$,o>,h; 


r\  -roo  r 

'Ab  L. 


h 

j=i 


4  TJ 


J 


or 


q 

+  (x,yo,Zp;0,*,a»»Tj)  =  V 

>1 


Type  VI  Model:  Constrained  Distributions 

As  previously  stated,  inequalities  which  are  specified  in  the 
object  model  describe  region  boundaries  of  th_  object  model  distribu¬ 
tions.  For  this  reason,  we  refer  to  these  distributions  as  "constrained 
distributions";  and  the  ordering  relationships  are  called  "constraints". 

'The  constrained  distribution  models  which  are  considered  here  may 
also  involve  structural  relationships.  Such  situations  are  dealt  with 
in  the  derivation  schemtic,  but  are  not  included  in  the  Type  VI  illus¬ 
tration. 


The  constraining  surface#  are  denoted  Y  *  G(x,®).  In  general  the 
requirement  that  parameters  ®  be  evaluated,  introduces  great  difficulty 
in  computation  with  numerical  methods  currently  available.  However, 
for  constant  surfaces  Y-®.  there  is  no  particular  trouble.  Also,  the 
problem  is  resolvt  i  completely  if  the  integration  can  be  performed 
analytically. 

Type  VII  Model:  Varifom  or  Discontinuous  Models 

For  models  of  this  type,  various  regions  of  the  model  space  are 
represented  by  constitut.  i.  elements  of  the  model  which  are  di6,’inct  in 
mathematical  form.  The  regions  are  separated  by  specified  com  training 
surfaces.  The  constraining  surfaces  in  this  ease  are  called  "shock 
.rents",  or  in  the  one -dimensional  case,  "shock  points".  For  example, 

<i  melting  point  or  a  boiling  point  is  a  shock  point;  in  meteorology, 
storm  fronts  or  air  mass  boundaries  are  shock  fronts. 
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TYPE  VI  MODEL:  CONSTRAINED  DISTRIBUTIONS 


Object  Model 

1.  Joint  Density  Function  ^(x,\':4) 

2.  Structural  Relationships  Z  -  F(x,y,0) 

3.  Constraints  y  -  Gfx.w) 

Calibre! ion  Model  ^(e,,ey,et| x,y,z;  0) 

Derivation  of  the  Observation  Model  Hypothesis 

1.  <*V  ~  ^(x0,y0,ze|x,y,z; 

2.  <Kxc,vo,*olx.ytZ,<&  -■  ^x0,y0l^o!x,y;  F) 

3.  i(x,y  0  •  4'(x,,y0tz0lx,y;  6,0)  .=  iMx0,y0,z0,x,y;  0,0,*; 

r  *'(*.  *>)  1 

4-  ^(xo.yo.2c.x.y^AOJy  dx  -  4(xo,yo,zo;0,O,*,u>) 
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Tvo  Type  VII  illustrations  are  used,  to  show  examples  of  both 
change  in  the  form  of  joint  density  functions,  and  change  in  the  form 
of  structuial  relationships.  A  more  compact  notation  than  that  of  the 
previous  model  type  has  been  adopted  in  the  derivation  schematic  for 
convenience. 

The  same  computational  difficulty,  as  in  the  previous  model  type, 
may  be  expected  here  if  the  cd  require  evaluation. 

Indirect  Observation 

First  consider  a  simple  measurement  situavion,  such  as  that  of  the 
lype  I  model;  except  that  one  variable,  which  is  not  subject  to  obser¬ 
vation,  with  or  without  the  influence  of  errors  of  observation,  is 
determined  only  as  a  specified  single-valued  function  of  the  remaining 
variables.  Under  this  hypotnesis,  all  of  the  variables  assume  constant 
values,  the  determination  of  which  is  the  object  of  our  effort;  but 
specifically  we  wish  to  obtain  that  numerical  Value  which  is  "only  in¬ 
directly  observable". 

It  is  helpful  to  refer  to  the  Type  I  illustration  and  schematic,  in 
observing  that  no  amount  of  observation  can  provide  information  about 
the  single-valued  functional  relationship.  'Hie  functional  form  must 
be  specified  completely  in  the  initial  hypothesis.  However,  estimators 
of  the  hypothesized  constant  values  of  those  variables  subject  to  ob¬ 
servation  (and  also  error  of  observation)  are  obtainable  through  appli 
cation  of  the  Type  I  model.  But  moBt  important,  it  is  kn cnn  that 
single -valued  functions  of  maximum  likelihood  estimators  are  also 
maximum  likelihood  estimators.  Consequently,  the  desired  maximum  like¬ 
lihood  estimator,  for  that  constant  not  subject  to  observation  with  or 
without  error,  is  obtained  indirectly  by  simple  substitution  of  the 
available  estir  itors  into  t.he  completely  specified  single-valued  func¬ 
tion. 

Similar  considerations  apply  in  mapping  distributions,  one-to-one 
or  many-to-one,  from  spacec  of  which  all  vai  iables  are  subject  to 
observation  with  or  without  • rror;  onto  spaces  the  variables  of  which 
include  one  or  acre  which  are  only  indirectly  obser.'able.  The  ordinary 
transformation  of  coordinates  is  Invoked,  and  parameters  of  the  new 
dj ftribution,  which  are  determined  thereby,  retain  the  maximum  likeli¬ 
hood  properties  of  the  primary  estimators.  However,  the  equations  of 
transformation  must  be  completely  specified. 

Other  Model  Types 

T1  .  previously  discussed  model  types  no  doubt  encompass  a  signifi¬ 
cant  fraction  of  observational  situations  likely  to  arise;  however, 
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TYPE  VII  MODEL:  VARIFORM  OR  DISCONTINUOUS  MODELS 


Object  Model  (For  simplicity,  only  two  forms  arc  indicated.  1 

Form  A  1.  Joint  Marginal  Density  Function  (X;  ) 

2.  Structural  Relationships  y  -  FA  ) 

Fo'm  B  1.  Joint  Marginal  Density  Function  (X,i„  > 

2.  Structural  Relationships  y  FH(X,0(t) 

Shock  Front  fin  general  more  than  one  required.  )  X2  -  ,Ji 

Calibration  Model  ;(ex,ey |  x,y;  C) 


Derivation  of  the  Observation  Model  Hypothesis 

1.  4(e,tey  !  x,y;  0)  -  v  (x0,y0  lx,y;  <t) 

2.  A  <i(x0,y0lx,y;  ^  ~  <Kx0,y0lx;  0A  ,<J>;  FA  ) 

2. B.  <i(x0,y0lx,y;  4>)  —  *(x0,y0|x;  0„  ,<*>;  F„ ) 

3. A.  <JA  (x;  4v)  •  </A  tXo,y0  !x;  0A  ,<t>)  =  4A  (x0,y0,x;  0A  ,4>,i  A  ) 

•LB.  </  h  (X;  4h)  ■  4„(x0,y0/x;  0n,<P^  -  4  ^  (xq,j  o>x;  n  ) 

4.  Fl/"" 

L 

r  ♦  ®  r  ] 

♦  I  I  (x0.y0.xl.x2i  A*B  >dx2  v(X0ly0,  f>A  4a  ,U') 

- "  LG  (*i 
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many  situations  can  arise  to  which  the  available  model  types  do  not 
readily  conform.  For  example,  the  various  models  may  be  combined  in 
configurations  of  great  complexity;  also,  it  will  probably  be  expedient 
to  select  certain  special  cases  for  particular  emphasis.  To  a  lesser 
extent,  it  may  be  useful  to  strive  for  greater  generality.  Hopefully, 
the  number  of  available  useful  model  types  will  grow  continually  as 
experience  is  gained  in  application  of  the  system. 

In  a  large  class  of  observational  situations,  not  ill  events  which 
are  potentially  observable  are  actually  observed.  Thus  in  general, 
there  exists  an  "attenuation  function"  or  "efficiency  of  observation 
function",  <*,  defined  over  the  space  of  object  variables.  In  such 
cases,  the  object  variable  joint  density  function  becomes 

*''00  c(x)>*,(x) 

As  a  final  remark,  in  combining  the  various  model  types  into  more 
complex  configurations,  whenever  possible  it  is  better  to  evaluate 
parameters  of  all  constituent  hypotheses  at  the  same  time,  in  a  single 
observation  model. 


PROBLEMS  IN  COMRJTATTON 


So  far  we  have  carefully  sidestepped  the  possible  problems  of  com¬ 
putation.  In  construction  of  the  observational  model,  difficulties 
can  arise  in  nonlinear  transformations  and  projections  requiring  inte¬ 
gration.  Following  construction  of  the  observation  hypothesis  and  the 
likelihood  function,  the  problem  of  maximising  log  L  is  encountered. 

With  respect  to  the  problem  of  integration,  presently  available 
methods  appear  to  be  inadequate  for  the  task  at  hand.  The  most  useful 
system  applications  will  probably  involve  integration  over  regions  of 
ir<a.<y  dimensions.  Classical  methods  are  accurate  but  require  excessive 
computation  in  the  higher  dimensional  cases.  Monte  Oarlr  methods  are 
probably  too  imprecise  for  most  system  applications. 

For  maximizing  the  likelihood,  there  are  several  useful  techniques 
available;  principally  the  Newt on- Ha phs on  method  for  aolution  of  simul¬ 
taneous  nonlinear  equations,  and  direct  search  methods  such  as  the 
method  of  steepest  ascent.  However,  it  is  more  probable  that  new,  more 
powerful  methods  will  be  required. 


It  is  our  stated  purpose  to  employ  automatic  computing  machines  for 
most  expeditious  application  of  the  system  techniques .  Thus,  although 
analytic  solutions  may  be  feasible  in  some  cases,  numerical  methods  are 
to  be  preferred  for  the  generality  of  their  application  in  automatic 
computation.  However,  application  of  numerical  methods  for  integration 
in  the  model  derivation  and  for  maximization  of  the  likelihood  imposes 
the  requirement  that  the  two  (or  more)  techniques  must  be  effected 
simultaneously.  This  compounds  the  difficulties  considerably. 

TSiese  considerations  have  led  this  author  into  an  investigation  of 
numerical  methods  suitable  for  use  in  the  system  applications.  The 
basic  idea  is  to  partition  the  multi -dimensional  region  of  interest  by 
a  series  of  cuts ;  As  the  cutting  proceeds  the  region  is  represented 
by  a  number  (usually  large)  of  subregions.  Prior  to  each  cut,  the  sub- 
region  to  be  cut  and  the  direction  of  cutting  are  selected  according  to 
appropriate  indices.  Alternatively,  an  index  for  termination  of  cut¬ 
ting  is  used  in  place  of  the  subregion  selection  index.  The  resulting 
subregions  are  then  represented  by  a  selected  point  set.  Operations 
on  this  representative  point  set  can  then  be  performed  to  effect  the 
solution  of  a  wide  variety  of  multivariable  numeiical  analysis  problems, 
Including  integration  and  direct  search  optimization. 

The  principal  work  to  date  has  been  in  the  area  of  multiple  inte¬ 
gration.  The  problem  of  non -variables -separable  functions  has  been 
resolved  by  selecting  as  the  representative  point  for  each  subregion, 
the  intersection  of  (n-l) -dimensional  mean  value  surfaces,  where  n  is 
the  dimensionality  of  the  region  of  integration.  These  methods  have 
met  with  considerable  success.  The  method  of  integration  is  presently 
programmed  for  an  IBM-704  computer  with  a  4000-word  magnetic  core 
storage.  Multiple  Integrations  can  be  performed  for  functions  of  up 
to  ten  variables.  Por  example,  the  integral  over  the  unit  interval  of 
Kxp(x1»X2*X3.xj^)  has  been  obtained  to  better  than  six  significant  digits 
of  precisian  in  about  seven  minutes.  Por  this  result,  32,455  points 
and  1,930  subregions  were  required.  Considerable  improvement  is  expec¬ 
ted  if  more  core  storage  is  available;  in  particular  with  respect  to 
increasing  the  speed  and  the  upper  limit  on  the  number  of  variables. 

Vlth  respect  to  the  problem  of  simultaneous  integration  and  maximi¬ 
sation,  developmental  work  is  in  progress.  The  approach  used  is  to 
generate  a  representative  set  of  points  as  above  but  over  a  higher 
dimensional  space,  which  Includes  as  subspaces  both  the  space  of  vari¬ 
ables  over  which  integration  is  to  take  place  and  the  parameter  space 
aver  which  maximization  of  the  likelihood  is  to  take  place.  In  the 
parameter  space,  the  selected  index  for  cutting  is  the  likelihood  value 
or  its  logarithm.  Subregion  midpoints  may  be  used  as  representative 
points  in  this  space.  Integration  is  best  held  to  relatively  low  pre¬ 
cision,  except  for  points  in  the  Immediate  neighborhood  of  likelihood 
maximum  points . 
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For  the  purpose  of  exploring  multi dimensional  surfaces,  such  as  the 
likelihood  surface,  a  representative  point  set  is  particularly  valuable. 
Localized  maxima  or  sa  Idle  points  are  not  likely  to  be  mistaken  for  the 
absolute  maxima.  Also,  singularities  and  discontinuities  tend  to  be 
readily  detected.  Such  a  characterization  of  the  likelihood  surface 
should  probably  be  reported  out  as  auxiliary  information.  It  could 
find  use  in  future  decision  theoretic  applications  of  the  model  as 
developed. 

In  any  case,  it  is  hoped  ttat  the  capability  for  solution  of  the 
system  computation  problems  will  soon  be  available.  This  author's 
work*  is  in  preparation.  Perhaps  the  main  point  to  be  made  at  this 
time  is  not  that  the  methods  proposed  here  for  computation  will  neces¬ 
sarily  be  those  which  are  ultimately  best  for  the  general  system  appli¬ 
cation,  but  that  whatever  methods  are  employed,  they  must  be  based  on 
computer-oriented  mathematics. 

Specifically,  the  methods  to  be  used  in  computation  must  not  be 
based  on  pencil  and  paper  mathematics.  It  is  inconceivable  that  inves¬ 
tigators  can  ever  hope  to  deal  with  problems  at  a  practical  level  of 
complexity,  if  they  are  to  be  restricted  to  the  present-day  pencil  and 
paper  analytic  techniques.  One  is  led  to  believe  that  a  new  kind  of 
mathematical  analysis  must  evolve,  which  is  exclusively  devoted  to 
c cmpute r - oriented  mathematics. 

Also,  if  our  goal  is  to  see  our  new  mathematical  methods  actually 
put  to  use  on  a  broad  scale,  it  is  necessary  that  most  of  the  labor  in 
application  be  removed.  Investigators  generally  do  not  aspire  to  be 
also  mathematicians.  They  will  not  be  disposed  to  accept  and  apply 
elaborate  new  procedures,  unless  they  are  also  attractively  packaged 
and  automated. 


UN  IDENTIFIABI  LI  T5f 


Throughout  this  paper  we  have  emphasized  the  Basic  Principle  of  pro¬ 
jection  Into  the  observable  variable  space.  At  this  point,  we  consider 
the  consequences  with  regard  to  that  information  which  la  necessarily 
lost  in  projection. 

In  the  usual  practical  situation,  it  is  not  possible  to  uniquely 
determine  (or  estimate)  some  subset  of  the  real  event  model  parameter 

♦See  "A  Methodology  for  Numerical  Analysis  of  Functions  of  Many  Vari¬ 
ables  ,  with  tt.’Tphasis  on  Multiple  Integration",  by  J.  W.  Hendricks. 
(Technical  Report  to  be  published'. 


values.  A  parameter  is  said  to  be  "identifiable"  if  and  only  if  a 
unique  value  or  its  consistent  estimate  is  obtainable,  given  full  know¬ 
ledge  of  the  distribution  of  observations.  In  general,  unidentif labil¬ 
ity  is  due  to  the  loss  of  information  in  projecting  the  distribution  of 
real  events  into  the  apparent  event  space  of  observable  variables.  How¬ 
ever,  in  trivial  instances,  a  case  of  unidentif lability  in  effect  can  be 
caused  by  specification  of  superfluous  parameters. 

The  concept  of  identification  may  be  more  clearly  explained  by  means 
of  an  example.  In  considering  the  error  equations,  it  is  evident  that 
for  each  observed  event  x^,  there  is  an  infinite  class  of  possibly  true 
event  pairs  (x,ex)  which  could  serve  to  explain  the  observed  event.  It 
follows  that  for  any  given  distribution  of  observed  events  ^(xq),  there 
is  an  infinite  class  of  joint  density  functions  f(x,ex)  which  could  have 
served  to  generate  \|r(xo)  •  In  general,  in  attempting  to  identify  the 
true  Joint  density  function  ♦*(x,ex)>  we  may  or  may  not  be  concerned 
with  parameter  evaluation. 

However,  in  this  system  we  are  committed  to  parametric  representa¬ 
tion  of  the  initial  hypotheses.  The  functional  forma  hypothesized  for 
♦(x,6)  and  <Kex/x;t)  serve  to  restrict  the  infinite  class  of  density 
functions  \^(x,ex;6,^)  from  which  the  given  ^(xq;0.$)  could  have  been 
generated.  Individual  members  of  the  class  are  identified  by  unique 
values  of  the  6  and  4.  To  state  that  the  true  Joint  density  function 
♦*(x»ex;0*-**)  identifiable,  is  to  state  that  the  true  parameter 
values  0*  and  4*  are  all  individually  identifiable. 

Let  the  9*  and  4*  be  exclusively  location  or  translation  parameters 
(i.e.,  not  shape  parameters)  of  a  true  structural  relationship  F*,  and 
of  an  associated  conditional  error  distribution  respectively.  These  0* 
and  ♦*  are  particularly  susceptible  to  unidentif  lability  for  the  reasons 
given.  Consider  the  situation  of  Figure  3,  where  a  large  error  bias, 
uniformly  applied  over  the  x,y  space,  is  indicated.  Observe  that  errone¬ 
ous  values  of  0, namely 0’  in  the  illustration  are  also  compatible  with 
the  observations  under  the  assumption  of  little  or  no  translation  (or 
bias)  of  the  error  distribution. 

In  such  a  situation,  the  "*lse  hypothesis  F*  would,  for  the  indi¬ 
cated  observations,  actually  represent  an  equal  likelihood  alternative 
to  the  true  structural  relationship  F*.  F*  and  F'  are  of  course  only 
two  examples  of  the  Infinite  family  of  structures  which  would  represent 
equal  likelihood  alternatives  in  this  case. 

Bias  of  course  Is  not  necessarily  constant  over  the  space;  there  may 
be  more  complex  interaction.  Also,  it  is  not  only  the  location  para¬ 
meters  which  interact.  However,  unidentif lability  associated  with  inter¬ 
action  between  the  object  phenomenon  and  the  error  phenomenon  is  but  one 
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INTERPRETATION  OF  RESULTS 


Surfaces  of  Unidentif lability 

Ilius  the  question  of  identif lability  arises  for  models  generally, 
and  in  particular  for  individual  parameters.  Unidentif inbility  is 
evidenced  by  the  fact  that  the  likelihood  function  in  such  cases  pos¬ 
sesses  no  unique  maximum.  However  in  general,  all  pertinent  parameter 
information  is  contained  in  the  relationship  between  parameters  wliich 
is  implied  by  the  locus  of  equal  likelihood  maximum  points  in  the  para¬ 
meter  space,  namely 


L(e,«) :«  l 


Thus  at  least,  the  functional  relationship  between  individually  uniden¬ 
tifiable  parameters  is  always  identifiable  in  this  sense. 

Loci  of  likelihood  maximum  points,  here  called  "surfaces  of  uniden¬ 
tif  lability",  are  likely  to  be  multi -dimensional  surfaces.  For  simpli¬ 
city  in  discussion,  consider  only  nonotonic  surfaces.  Regardless  of 
the  dimensionality  of  the  parameter  space,  the  dimensionality  of  such 
a  surface  is  equal  to  the  number  of  parameters  which  must  be  evaluated, 
arbitrarily  or  otherwise,  in  order  to  establish  a  unique  maximum  like¬ 
lihood  point  in  the  parameter  space.  However,  it  is  neither  necessary 
nor  desirable  that  parameter  values  be  assigned  arbitrarily,  thereby 
imposing  information  which  is  unjustifiei  by  either  observation  or 
theory. 

It  should  be  remembered  that  our  ultimate  goal  is  model  development. 
To  the  extent  that  non-unique  parameter  estimates  constitute  an  unsatis¬ 
factory  conclusion  to  the  investigation,  the  investigator  may  wish  to 
modify  his  specified  hypotheses  or  take  some  other  appropriate  action. 
Ails  he  is  uniquely  qualified  to  do.  He  alone  may  possess  valid  auxili¬ 
ary  information,  not  previously  included  in  the  specifications;  and  he 
alone  establishes  the  goals  and  criteria  of  the  investigation.  For 
example,  he  may  know  of  no  mechanism  Involved  in  observation  which 
could  justify  the  presumption  of  bias  (E(ex)*0).  He  might  therefore 
choose  to  assume  no  error  bias  (using  a  criterio*  of  maximum  simplicity) 

A  system  requirement  of  no  error  bias  would  tend  to  dispose  of  un¬ 
identif  lability  due  to  interaction  between  the  calibration  and  object 
models.  However,  it  is  specifically  not  the  object  of  this  discussion 
to  require,  or  even  investigate,  conditions  for  complete  identif lability 
of  either  models  or  individual  parameters.  In  no  case  is  it  reasonable 


that,  for  computational  or  analytic  convenience,  ve  impose  assumptions 
and  conditions  which  are  not  realistic  or  generally  acceptable  in 
scientific  investigation. 

Thus,  Ciiaracterization  of  the  set  of  all  equal  likelihood  maximum 
points  in  the  parameter  space  is  the  appropriate  ultimate  conclusit.i 
in  any  single  application  of  this  system.  Further,  it  is  essential 
tint  the  computer  output  be  of  such  a  form  as  to  facilitate  the  investi¬ 
gator's  interpretation  and  use  of  the  results.  The  characterization 
can  take  the  foru  of: 

1.  A  unique  evaluation  or  estimate  for  each  parameter. 

2.  A  set  of  functional  relationships  between  parameters  which  are 
individually  unidentifiable. 

3*  A  representative  subset  of  the  likelihood"  maximum  points. 

4.  A  combination  of  these. 

With  respect  to  computer  output,  in  sane  simple  cases,  tabulation  may 
suffice.  In  other  cases,  graphical  or  analytic  representation  of  the 
surface  of  unidentif lability  would  be  required.  Analytic  representation 
might  possibly  require  "fitting"  a  specified  functional  form  to  a  repre¬ 
sentative  set  of  points  either  in  reapplication  of  the  system  or  by 
appropriately  modifying  the  initial  hypotheses.  However,  graphical 
representation  of  selected  cross-sections  would  seem  also  to  be  gener¬ 
ally  convenient. 

In  some  trivial  cases,  superfluous  unidentif iaole  parameters  can  be 
eliminated  by  simple  substitution,  using  the  indicate  relationship. 
However  in  general,  more  elaborate  procedures  are  ret  ilred;  and  in  acme 
cases,  the  unidentifiability  will  probably  never  be  satisfactorily  re¬ 
solved  .  . .  . 

Calibration 

Presumably,  the  moat  cannon  form  at  unidentifiability  is  that 
associated  with  errors  in  observation.  In  measuring  a  table  with  a 
previously  unused  ruler,  we  cnn  never  be  absolutely  sure  that  the  scale 
is  not  significantly  either  too  short  or  too  long,  and  if  so  by  how 
much.  Classically  the  scale  is  calibrated  against  a  known  and  accepted 
standard.  It  clearly  serves  no  purpose  to  check  the  measurement  against 
other  previously  unused  scales,  assuming  no  other  information.  Or  in 
general,  when  calibration  models  include  unevaluated  location  parameters, 
no  purpose  is  served  in  substituting  other  methods  of  observation,  the 
associated  calibration  models  for  which  also  involve  unevaluated  loca¬ 
tion  parameters. 


Actual  calibration  against  appropriate  "standard  object  phenomena" 
is  much  to  be  desired.,  perhaps  more  so  for  the  more  complex  phenomena 
and  models.  In  application  of  the  system  for  the  purpose  of  calibration 
we  seek  more  complete  specification  of  the  calibration  model  prior  to 
its  use  in  conjunction  wi+h  other  nonstandard  object  uodels.  The  idea?, 
standard  object  phenomenon  might  consist  of  a  finite,  more  or  less  uni¬ 
form  lattice  of  event  points  distributed  over  the  object  variable  space, 
at  least  encompassing  the  range  of  variables  within  which  the  object 
phenomena  to  be  investigated  are  likely  to  be  centered.  Rararoeter 
values  of  the  standard  object  phenomenon  mciej  are  most  appropriately 
established  by  definition  or  with  reference  to  other  known  standards. 

Thus,  in  the  usual  calibration  situation,  the  calibration  m  del 
parameters  alone  remain  to  be  evaluated.  Unidentif lability  due  to 
error  is  thereby  eliminated,  unless  there  is  unidentifiability  inherent 
in  the  error  phenomenon  model  as  specified.  However,  it  is  conceivable 
that  not  all  standard  object  phenomenon  parameter  values  will  be 
established  previously.  In  such  a  case,  they  may  be  evaluated  along 
with  the  calibration  model  parameters,  providing  no  unidentifiability 
is  introduced  by  the  interaction. 

A  requirement  that  actual  calibration  be  employed  in  all  cases  may 
be  unnecessarily  restrictive.  Giver  a  surface  of  unidentifiability  or 
the  set  of  all  maximum  points,  in  general  we  desire  that  the  appropriate 
number  of  parameter  values  necessary  for  complete  identification  be 
included  in  the  model  specifications.  This  does  net  mean  that  all 
parameters  evaluated  by  specification  must  be  calibre  cion  model  para¬ 
meters.  They  need  only  constitute  that  subset  of  the  necessary  oize 
about  which  the  most  information,  not  otherwise  employed,  is  available. 
Actual  calibration  is  only  one  possible  source  of  such  inforration. 

In  the  usual  practical  application  of  the  system,  where  unidentifi¬ 
ability  due  to  error  is  present,  actual  calibration  is  probably  to  be 
preferred  over  other  means  of  effecting  complete  identification.  In 
each  specific  instance,  only  the  investigator  is  qualified  to  decide. 

Considerations  in  Ranking  Models 

An  investigator  may  choose  initially  to  include  an  additional  para¬ 
meter  in  the  model  (a  "fudge  factor"),  hopefully  to  counteract  a  poten¬ 
tial  weakneso  in  his  theory,  or  as  a  device  in  theoretical  exploration. 
The  introduction  of  additional  parameters,  above  and  beyond  those  which 
are  truly  justified  by  current  theory,  will  not  necessarily  result  in 
unidentifiability.  Parameters  which  are  superfluous  in  this  sense  may 
take  on  estimated  values  which  are  unique  but  effect  no  significant 
influence  in  the  model.  Another  possibility  is  that  tne  additional 
flexibility  thereby  introduced  into  the  model  will  result  in  a  higher 
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but  misleading  value  for  the  maximum  likelihood;  a  better  fit  is  ob¬ 
tained  to  that  variation  in  the  observations  which  is  attributable 
solely  to  the  random  component  in  sampling.  Since  this  is  always  a 
dangerous  possibility,  the  need  for  large  samples  and  verification  is 
again  evident. 

A  deficiency  in  the  number  of  specified  unknown  parameters  may  or 
may  not  result  in  unidentif lability .  However,  such  a  deficiency  always 
constitutes  a  specification  error  and  a  poor  fit  of  the  model  to  the 
observational  data  must  result,  with  a  consequent  low  value  for  the 
maximum  likelihood. 

In  general,  we  can  expect  a  higher  maximum  value  for  the  likelihood 
with  a  more  complicated  model;  but  as  always,  we  do  net  further  our 
goal  of  r  jdei  development  unless  a  state  of  control  is  also  .'stablished. 
Per ha os  the  maximum  likelihood  values  should  also  be  considered  in  re¬ 
solving  questions  of  state  of  control. 

Heca  3.1  also  that  maximum  likelihood  is  an  acceptable  criterion  for 
selection  of.  the  best  model  from  o  number  of  models,  providing  that  the 
same  data  is  used  in  each  case.  Thus  a  direct  maximum  likelihood  com- 
pe risen  of  ooject  phenomenon  mod ;ls,  tfiich  have  been  developed  using 
different  methods  of  observation,  is  not  necessarily  valid.  Additional 
information  concerning  a  state  of  "mutual  control"  on  the  designated 
phenomena  may  be  required.  Also,  an  adjustment  must  be  made  for  the 
(inference  in  sample  size. 

Availability  of  the  likelihood  value  makes  its  use  convenient  in 
ranking  models ;  also  the  associated  difficulties  do  not  seem  to  be  too 
Berlova.  However,  this  method  of  ranking  is  probably  not  that  which 
is  ultimately  best.*  Consider  that  the  ability  to  predict  has  been 
associated,  in  ar.  earlier  section,  vith  the  idea  of  understanding. 
Prediction  la  only  temporal  extrapolation.  Thus  ve  can  regard  the 
ability  to  extrapolate  fn  the  direction  of  other  variables  as  a  useful 
criterion  of  the  achievement  of  understanding,  and  consequently  a  useful 
criterion  in  ranking  models. 

Every  model,  and  in  fact  every  functional  model  component,  can  be 
regarded  as  an  implicit  function  of  any  variable  in  nature  which  is  not 
already  involved  in  the  functional  expression.  Model  components,  by 
their  specification,  are  presumed  to  be  unchanging  in  the  coordinate 
directions  of  those  variables  which  are  omitted.  Thus,  no  variable 
la  excluded  from  consideration  in  this  senae. 

•Note  that  "Goodness  of  Fit",  in  its  specialized  sense,  is  not  con¬ 
sidered  in  this  discussion. 
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Clearly,-  the  ability  to  predict  is  also  evidence  of  a  state  of 
control;  and  further,  the  ability  to  extrapolate  in  the  direction  of 
other  variables  is  evidence  of  a  generalised  state  of  control.  Thua, 
given  some  quantitative  measure  of  generalised  state  cf  control,  such 
measure  my  also  be  appropriate  if  not  superior  to  likelihood,  in 
ranking  models . 

State  of  Control 

Rec ill  that  the  object  phenomenon,  as  designated,  actually  consti¬ 
tutes  a  class  of  phenomena,  namely  that  which  is  the  logical  inter¬ 
section  of  classes  established  by  the  individual  designating  statements. 
Within  the  designated  class  intersection,  there  is  generally  room  for 
some  detectable  variation  (l.e.,  acceptable  variation  In  "representa¬ 
tive"  sets  of  observations),  random  or  otherwise.  Any  6uch  variation 
which  is  excessive,  according  to  some  criterion,  is  by  definition  evi¬ 
dence  of  lack  of  control. 

In  any  serious  attempt  ut  model  development,  an  indicated  signi¬ 
ficantly  large  lack  of  control  can  hardly  be  tolerated.  The  investiga¬ 
tor  will  usually  chooee  not  to  narrev  the  scope  of  his  investigation. 
From  the  information  available,  he  will  probably  attempt  to  adjust  his 
initial  specifications.  Parameters  will  tend  to  vary  in  the  degree  to 
which  they  are  reproducible.  He  may  therefore  specifically  devote. his 
attention  to  those  model  parameters  the  values  of  which  are  relatively 
out  of  control. 

Criteria  for  control  are  beat  established  with  reference  to  the 
distribution  of  observations.  However  as  stated,  the  acceptable  resi¬ 
dual  variation  la  also  Influential  in  the  parameter  space.  Consider  a 
situation  involving  unidentifiability,  wherein  a  number  of  independent 
Investigations  are  conducted;  and  in  each  case,  with  the  same  object 
phenomenon,  the  same  method  of  observation,  and  the  same  initially 
•pacified  hypotheses.  Then  from  eacn  investigation,  a  surface  of  un- 
ldant  If  lability  results. 

Assuming  only  one  member  In  the  designated  object  class,  or  a  "per¬ 
fect  state  of  control",  for  large  sample  slee  we  would  expect  that  all 
of  tbs  surfaces  would  effectively  coincide.  However,  in  practical  model 
development,  even  assuming  no  sampling  error,  perfection  is  not  likely 
to  be  achieved.  A  more  likely  result  is  depicted  in  Figure  4.  Here, 
four  surfaces  I*,— — ,E^  are  shown  to  be  intersecting;  although  in 
general,  there  is  no  rsason  that  they  should. 

Ir  the  parameter  space,  measures  of  state  of  control  could  refer 
to  variation  about  some  "central  point".  For  example,  the  least  squares 
point  for  distances  normal  to  each  surface,  where  such  exists,  might 
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Fig.  4  Evidence  of  Specification  Error  or  lack  of  Control 


be'' optimum.  Any  decreasing  function  of  the  minimum  sum  of  squares 
could  then  be  used  to  '.ndicate  degree  of  control. 

The  dotted  circle  of  the  illustration  can  be  said  to  define  a 
region  of  acceptable  risk;  and  in  the  same  sense,  the  least  squares 
point  may  be  considered  to  indicate  the  safest  parameter  estimates. 

This  is  a  barefaced  behavioristic  point  of  view;  but  it  seems  to  be 
reasonable  under  the  circumstances* 

Thus,  in  these  ways,  the  development  of  mathematical  models  may  be 
pursued. 


CONCLUDING  REMARKS 


Ve  have  been  primarily  concerned  with  the  elimination  of  undue 
restrictior  s  vhich  are  imposed  on  the  investigator  in  his  acceptance 
of  currently  available  procedures  for  data  analysis;  a  corollary  concern 
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has  been  to  point  out  cane  important  deficiencies  of  ordinary  investi¬ 
gative  procedures.  To  this  end,  an  extension  and  synthesis  of  scienti¬ 
fic  methods  has  been  attempted. 

A  subsidiary  coal  has  been  to  contribute  to  the  elimination  of 
inefficiency  in  scientific  investigation  through  contribution  to  methods 
of  organization  and  presentation.  Ultimately,  automation  must  effect 
the  greatest  influence.  In  view  of  these  goals,  it  is  hoped  that  this 
paper  may  also  serve  as  a  procedural  guide  or  handbook. 

By  the  term  "initial  specifications",  it  is  not  intended  to  imply 
that  no  labor  is  involved  In  their  formulation.  In  fact,  it  is  likely 
that  this  will  constitute  the  investigators  most  difficult  problem, 
since  it  is  here  that  nis  unique  professional  skills  find  their  great¬ 
est  application,  however,  it  is  here  also  that  the  statistician  may 
prove  most  helpful.  In  particular,  he  may  assist  in  or  even  direct 
construction  of  the  calibration  model.  Also,  he  may  assist  in  the 
actual  calibration. 

The  development  of  mathematical  models  must  of  course  proceed  con¬ 
currently  with  the-  development  of  the  associated  scientific  discipline. 
In  the  usual  situation,  initial  hypotheses  ere  formulated  with  refer¬ 
ence  to  previously  developed  object  models,  and  m  the  light  of  new 
information.  Consequently,  no  overwhelming  difficulty  need  attend  con¬ 
struction  of  the  initial  hypotheses  at  any  one  Btage  of  the  development. 
The  same  conditions  of  course  apply  in  formulation  of  the  calibration 
model  typotheses. 


REFERENCES 


1.  3rai.thvBi.te,  R.  B.  (1953)-  Scientific  Explanation,  A  Study  of  the 
Function  of  Theory,  Probability  and  Lav  in  Science.  Cambridge,  at 
the  University  Press. 

2.  Cramer,  H.  (1946).  Mathematical  Methods  of  Statistics.  Princeton 
University  Press. 

3.  Denting,  W.  E.  ( .'943) .  Statistical  Adjustment  of  Data.  J-.hr.  Wiley 
and  Sons . 

4.  Fisher,  R.  A.  \’l950).  Statistical  Methods  for  Research  Workers. 
Hefner  Publishing  Company. 

5.  Haavelmo,  T.  (1944) .  The  Probability  Approach  in  Econometrics. 
Econometrics,  Vcl.  12,  July  Supplement. 

6.  Kendall,  M.  G.  (1951)-  Regression,  Structure,  and  Functional 
Relationships.  Part  I.  Biometrlka  38:11-25. 

_ _ _  (1952)  Part  II.  Biometrika  39:96-108. 

7.  Koopmans,  T.  C.  and  Reiera^l,  0.  (1950).  The  Identification  of 
Structural  Characteristics.  Annals  of  Mathematical  Statistics 
21:169-101,  June. 

8.  Medana Ky,  A.  (1959).  The  Fitting  of  Straight  Lines  When  Both 
Variables  are  Subject  to  Error.  American  Statistical  Association 
Journal,  March,  pp.  173-306. 

9.  Mood,  A.  M.  (1950).  Introduction  to  the  Theory  of  Statistics. 
McGrav-HilI. 

10.  Moore,  R.  H.  and  Zelgler,  R.  K.  (i960).  The  Solution  of  the  General 
Least  Squares  Problem  with  Special  Reference  to  High  Speed  Computers 
Los  Alamos  Scientific  Laboratory  Report  IA-2367. 

11.  Munroe,  M.  E.  ( 1951)  -  Theory  of  Probability.  McGraw-Hill. 

12.  Shevhart,  W.  A.  (1939).  Statistical  Method  From  the  Viewpoint  of 
Quality  Control.  Department  of  Agriculture,  The  Graduate  School. 

13.  Trotter,  H.  F.  (1957)*  Gauss's  Work  (1802-1826)  on  the  Theory  of 
Least  Squares,  An  English  Translation.  Atomic  Energy  Carsnission 
Technical  Report  3049;  Statistical  Techniques  Research  Group, 
Princeton  University. 


53 


